The Second Age is already the most obscure era of the setting (barring the Fourth I guess) so it would indeed be weird to hamstring them even more. Plus the setting won't be that recognizable; there's no Rohan at all, no Gondor or Arnor until the closing days of the Age, nothing happens in the Shire (which is how the Hobbits like it, obviously, but still), the Dwarves and Elves get along fine in Eregion, Moria isn't a shithole yet, the Ents are busy herding trees, so on and so forth. Numenor would be interesting to explore but it's the only part of the Age that actually IS covered fairly extensively by the Appendices so we're SOL there as well I guess.