I farm ghost ore all the time around 5 or 6am when I can't sleep. I think I got around 700 ore the other morning...nothing happened to me, nothing has ever happened to me. I think they need more than just seeing a character going along a farming path and seeing them get a lot of mats before they drop the ban hammer.