Silvan mixed with others in the Second Age because of refugees from Beleriand. Those refugees even ended up eventually becoming leaders or "high up" because they mixed and even brought "normal" elven language back to the group. It doesn't really break the lore for a watchpost to have all Silvan elves. Even if it is just a small group that answers to the High King. As for traders or other people noticing it is simple. They aren't many. It has only been a few weeks at most and the Orcs could be killing others. We don't know where on the routes the watchpost is located.