It's not supposed to be polite, it's just a way of saying "not interested but I still want to keep you around to talk in college and maybe you can sometimes buy me a drink after or help with some assignment". If I just directly told them I wasn't interested in inviting them over (ever) or that I preferred my.. um, "real friends" for that or that they were just nice to talk to sometimes but I wasn't interested in anything further.. Well, they probably wouldn't even talk to me after that or I'd have to do a lot of apologizing later and blame some "mood". It's better to avoid that and just tell the little white lies, I think
I know it sounds kinda shallow or even bitchy but the truth is, you can only have so many friends you hang out with. Does it mean I should just stop talking to everyone once the limit is reached? Besides, if someone told you "I like talking with you in school to pass the time but don't want to be your friend or anything", how would you feel and could you have a friendly talk with that person again?