First of all, no, they were not hard in quest gear. Besides you were supposed to do normals before you went in heroics to get the gear "required" to do heroics.
Second of all, people left those dungeons because they didn't want to be bothered with a heroic that would take 20-45 minutes or more with a bad group when you could do azjol nerub for the same reward in 15.