Nasus is pretty crazy late game, but yea, the problem is with people trying to control him early so that there won't be a late game.
I think other than wave control and personal skill, the jungler will need to put in an extra effort to at least guard the Nasus which may mean less attention on other lanes or farming time in the jungle. If they manage to counter-jg (as in counter-gank) where Nasus gets 1 or 2 kills early, I think he'd still be pretty viable.
The problem comes when the Nasus has the mentality of "I'm just gonna farm till late game then come out and fight" and/or when the jungler thinks "I just need to snowball Nasus and he'll win the game for us", worse when both are thinking like that. Nasus has a pretty nifty slow, so he could actually have quite some use on TPs (and not just use it to TP back to lane to farm after a death).
It's easier on lower elos because not only that your opponents let you farm, map awareness is generally lower and the opponent jungler usually is more absorbed with farming the jungle rather than farming champions, so it's much easier to scale into late game compared to when people start to think which lanes to snowball fast so that they can take over the game.