I noticed my server is slowed by I believe it's not permanent. Some people will come back. I see some people leaving but they might be mistaken. I believe across the game there is a slow down.
Sure, if you go to a huge server with pugs at 4 o'clock in the morning - yes, those realms exist - it might not be seen, but even those would go from like 180% capacity to 120%; it doesn't mean they weren't slowed down. Well, one could claim they'd better be in a filled server than a regular one but not sure if it's always optimal. It probably needs a sweet spot.