You're effectively just saying "we're going to deviate anyways" and that skin colour isn't really important on a narrative level (which no one denied). But when you are portraying a world (or time period) that is different from our own that has its own implied history, groups of people etc. the suspension of disbelief becomes much easier when this is also reflected in the appearance of the people. Doesn't really matter if the world is fictional or not either. If I watch a movie about feudal Japan I'd expect the people there to look the part. It's not the kind of thing that makes or breaks a narrative but it's pretty much always going to be perceived as jarring when you don't adhere to it because it will send the message that you aren't taking the world building seriously.
This is basically just repeating what you said earlier. On a sidenote, what academics consider to be important to the analysis of text isn't really all that relevant when it comes to the question of whether the average person watching the show feels like the source material is receiving the proper respect and whether the adaptation achieves the sort of inner consistency they have come to expect from Tolkien's world.
Well, this is just an accusation of hypocrisy. There's no excuse to deviate in other areas (like not making Gil-galad's hair silver). It's just that skin colour will unfortunately always stick out more for obvious reasons and is usually associated with a whole cluster of things (like different hair, eye colour etc.) that is also loosely connected to things like geography, culture (like in the case of hairstyles). I think most people understand this on an intuitive basis.
I think what this really boils down to is that you're only coming from this from a purely dramatic perspective. You're just interested in seeing a good story and interesting characters. Other people are interested in being transported to a vibrant and authentic world that tries to be more than just a stage prop and takes itself seriously enough to hold up to scrutiny. In the case of Tolkien, the world building happens to make up a not insignificant part of the appeal. You don't care about that stuff? Fine. However, there are plenty of people who do.