There is a profound difference between a game and a visual novel. For something to constitute a game, it must ask something of the player in terms of strategy or skill (often both). There has to be some kind of struggle to be overcome, in some form or another. In the absence of these qualities, by contextual definition, something cannot be rightly classified as a game.
There are plenty of interactive mediums, and many of them are very good. One notable example is linked below, by Jester Joe: The Stanley Parable - It's extremely well-made. But there is a strange controversy over semantics in this respect, and while I'm intimately familiar with the debate, I still find it extremely vapid. While they are not mutually exclusive, the core concepts meant to be conveyed by either term are fundamentally distinct. The conflation of "game" and "visual novel" seems to be largely political, as though the former title has more status or merit. But this is silly. The quality of an artistic work shouldn't be determined by its style or medium alone.
Ultimately, people need to be intellectually honest about nuanced distinctions. Otherwise, why bother classifying anything at all? We could just call everything "squanch" and be done with it. Obviously, I have no issue with people calling an interactive story a "video game" in the causal sense, just to convey a basic meaning. But falsely conflating the concepts in all respects is just silly.