The "diversity" was less a direct problem and more a talking point for the showrunner prior to the release of the show. Rather than talk up how faithful he was going to be to the books, or how he couldn't wait for you to see X, Y, and Z, he played up the diversity angle which made fans of the book concerned. It could be done (frankly everyone from the Two Rivers was supposed to look quite tanned (both a bit of pigment and alot of time outside), so the lily white Matt comes off strange. Rand was supposed to be quite tan as well, for all his red hair and grey eyes, but they could have worked inside those lines. They didn't, and people were concerned what else they would change.
If it was just that and the actors were awesome and the show flowed well, people probably would have shut up and enjoyed it. However, most of the cast was bland to bad, or ruined by the changes made to the story. They ruined most of Emond's Field and made the town feel seedy and sketchy. They aged up the characters so they could start throwing sex in (which is pretty minimal in the books, especially at the start). They made Perrin both kill a wife he never had to "make him sad" and then made so little effort to pay off any of his story from book 1 that he may as well not have been in it. They made the focus of the story the girls, whether that was agenda or attempt to disguise who the dragon was... eh.... and then utterly removed any stakes from their world at the end of the season. They ruined Lan and made him whiny and a pump-and-dump champ. They completed ignored Caemlyn. They cast an old unattractive woman for Min who is never going to look good in tight pants.
Combine that with the stuff they added in, both characters and places that added nothing to the story but stole runtime hours, and you ended up with one of the most disappointing shows since GoT S8.
It did make me break the books out and start re-reading them, so it scores a point for that, but it makes the show an even paler World That Might Be.