Conclusion: Maw caps at 12 targets. Spreadsheet says 13 in screenshot but after eliminating the outliers circled in red that becomes 12. If you're in a 10-man on an ideal fight with favorable stacking I'll assume you can average 8 targets per proc when it matters. Then you'd be losing a third of maw's baseline HPS from 25-man (or from the other point of view: maw is 50% better in 25 than 10). I used to see cleansing flames coming in at 2500 HPS after overheal when I was doing 25, so in 10 man it should give you around 1600 HPS on favorable fights. Stats on heroic scalpel are worth on the order of 1k-1.5k HPS burst for heavy AoE (some fraction of that sustained).
Again it winds up to be a decision between controllable throughput and higher baseline HPS. Certainly in 10 there are fewer situations where you'll actually be able to get that cone with an average of 8 people standing in it, so having a macro to switch between the two would be ideal.