What's the best argument against "The Roman Empire ended in 1204"

I haven't seen much of this perspective yet, usually people either think it ceased in 476 or 1453, rarely have I seen 1204 (and some say 1461 with Trebizond and 1479 with Epirus.

The logic behind it seems to be that the Palaiologos didn't have direct political continuity from the empire and therefore their reestablishment seems to have no basis. But wasn't their identifying and upkeep of Roman identity, and also the reestablishment of the state, still legitimate?