Not Nero, but Tiberius, and Tiberius called them his "fishies".

Caligula did indeed come along right after, though.

Shit like this goes even further back, though, at least to the Greeks. Athenians especially made it a point to fuck little boys at or around the time of puberty, up to adulthood. Likely goes back a bit further still, but the records get spotty, or at least nobody bothered mentioning it.