Daemon mentioned how crime was rampant outside the castle. If you let it go unpunished for years and years I'm sure most people would know who the criminals are. The gold cloaks are not an outside force, they are recruited citizens tasked with policing their own city. I'm sure they have a good clue who to round up when set loose.