He's being honest with the one child he's had he thinks can be of use to him. He doesn't see silly little mortals as being worth much of anything, so he doesn't understand why Peter would get all that upset, especially given what Ego's offering him.
It may be "stupid", but it's because Ego is "stupid"; he lacks any capacity to understand the mortal perspective. It's a major character issue they drill into your head several times over in the course of the film.