Are variants in Loki from different universes?

It’s easy to forget that Hiddleston’s lead Loki is actually a variant, whose history diverges from the core version we saw die in Avengers: Infinity War – but with this main Loki variant arriving in an alt-universe TVA in the show’s final scenes, that idea may come back into stark focus in Loki season 2, if not before.

Are there different celestials in different universes?

But yes every universe has its own celestials. Usually they’re consistent. There’s a story where the First Firmament comes back from the place where universes go when they die and tries to reclaim Everything. The current Marvel multiverse is the 7th/8th Cosmos.