How NOT to use stateofdps.com and worldoflogs.com

[Endus]

Before I begin, let me be clear that this post is NOT a critique of either of these sites. Their data is useful and has a purpose, but it is being misused by many to discuss class balance, which it fundamentally cannot be used for. Worldoflogs.com never claimed it could, and is intended as much for in-house examination of logs to optimize your own gameplay as comparing with others, and stateofdps.com takes the time to admit as much in their FAQ;

These numbers are not even close to the true average H/DPS across all WoW players

Given that stateofdps.com is a statistical site, they understand why their sampling methods mean their data is not a true average. Unfortunately, apparently a lot of readers DON'T read the FAQ or understand the math behind statistical theory. For that reason, I'm going to review a few principles of statistical theory, and examine the biases inherent to their sampling.

Fair warning, math ahead. But if you're not willing to learn the math, you really shouldn't be talking about average DPS numbers, which are math.


Let's begin with the basics. If you take a random sample for most kinds of things, you'll find that there tend to be more examples closer to the average than there are further outside. This works for grades and test scores, IQ scores, DPS numbers, etc. Once you control for external factors (and we'll come back to THAT point), the results tend to group up into a nice graph shape called a "bell curve". As underbogba pointed out in the thread, this is a generality and there are exceptions, where you end up with multiple peaks, but we're just handling basics for now rather than more advanced statistical theory. It looks like this;



The Mean Score is the average. The curve is highest there because there are more scores at/near the average than anywhere else.
The Standard Deviation is a statistical term for how wide the spread is. I'm not going to get into the details of the math, but you can see the percentage in the chart, for how much of the sample is within that standard deviation's distance; +/- one standard deviation from the mean is slightly more than 2/3 of the data. The important point for our purposes is that if a data set has less variation, if the numbers vary less from the mean, the standard deviation gets narrower which means that bell curve gets thinner and more pointy. If the numbers vary more, it will widen out. The area under the curve, the number of samples, remains the same, but the distribution changes. Another image to show what I mean is below;



The red line would be a typical distribution.
The blue line is a tighter, narrower distribution
The orange line is a wider distribution.

And the green line is a distribution with a lower average. The first three have the same average. That's important.


Now, to the important stuff. Stateofdps.com uses the top 200 parses on WoL, which are in theory some of the top players in the game. Out of the millions of players, this means their data is way, way to the right on the bell curve. Look at that second chart again. Look at where the lines are touching the 0.0 line, on their right hand sides. That's the Stateofdps.com numbers. See how the blue is significantly lower than the orange? The problem is that both distributions have the same average. By only selecting the top numbers, stateofdps.com isn't presenting you the average, they're presenting you numbers that are based at least as much on the variation within a distribution as anything else.


But Endus, I don't care about the scrubs who do bad, I only care about top performers!

Ahh, but that's not what you're getting. Skill is obviously a factor. I'm not saying those top parses aren't skilled players. But skill isn't the ONLY factor. And that's the problem.


The problem is those other factors. To review a few of them quickly (and I make no claims that these are the only ones, but they're fairly immediately obvious);

• Sheer random variation. See the second chart again. Different distributions push their top end further out, even if the averages are the same. In WoW, this comes down to how lucky you got with crits, proc timing, boss targeting causing you to move, etc. Lucky streaks give some classes significantly more benefit than others. I'll get into some hard sim numbers to wrap this up if you want specific examples.
  
  
• History. While it only works off the last 2 weeks, if there have been hotfixes or patches in that time, it skews the numbers on stateofdps.com, whether it was a nerf or a buff. They try and correct for that, but there's also a more subtle effect; classes that were underperforming and got buffed would have been more likely to be benched or at least passed over for gear in favor of those who were performing better at the time. This is more true of those guilds for whom performance is all, which are also those guilds most likely to be topping the WoL charts. As a result, lower performers, even if they've been buffed to have a competitive average now, will often still be behind the gear curve.
  
  
• Player bias. This one's tricky. There are buffs like Focus Magic and Dark Intent which will tend to go to the top DPS who can benefit from them. This has the effect of pushing their DPS even higher. Classes that benefit less from those buffs or whose DPS is slightly lower (or even just perceived to be lower for the reasons I'm detailing now) won't get those buffs, and this mean the gap appears larger than it actually is, on their own merits. Elemental Shaman, for instance, don't work as well with either crit boosts or periodic damage procs as some other classes, meaning we're usually not the first priority for either.

There are probably other factors that I'm not listing, but those are at least the ones that jump out at me.


And now, some numbers, to give you an idea of what the above means. For these numbers, I will be running Simulationcraft, version 4.0.6-18, the most recent as of this writing. I will be simming the results with 10,000 iterations, Patchwerk style fights, without Focus Magic or Dark Intent but all other raid buffs.


Elemental Shaman: 26,739 DPS, +/- 2100 (7.87%)
Enhancement Shaman: 26,858 DPS, +/- 2098 (7.81%)

Those are our current simmed BiS numbers. They're pretty close to each other. The important parts are to consider the average, and the variation; higher percentage variation means the class is more reliant on lucky streaks to top their DPS. It also means they can push higher when they GET those lucky streaks.

For example, one of the most random specs in the game, Fire mages;
26858+
Fire Mage: 25,650 DPS, +/- 3819 (14.89%)

Some important points to note; their average DPS is simming lower. The variation, on the other hand, is MUCH higher. Since we've removed gear and player bias and skill as factors, the ONLY thing affecting this is random variation. The variation with a Fire Mage is almost double that of a Shaman of either spec. If we were to only look at the top numbers for each out of those 10,000 variations, similar to how stateofdps.com works, we'd get this, instead;

Fire Mage: 29,469 DPS
Enhancement Shaman: 28,956 DPS
Elemental Shaman: 28,839 DPS

As you can see, by weighting our sample to only look at the top parses, Fire Mages move from ~ 1200 DPS below Shaman to ~500 DPS above Shaman. Based on nothing but random variation, and by ignoring all the parses where Fire Mages were just not getting their procs and their DPS was in the bucket as a result.


This is why you can't use stateofdps.com to show average DPS numbers across classes. Using something like simulationcraft is a MUCH stronger tool for comparison. Shaman of both specs are simming out at slightly under the average right now, but close enough that we're competitive. That's not to say we're perfect; Enhancement is scaling poorly and likely to fall behind in 4.2 if it's not addressed, and we're slightly under the average; we could use a small boost. Elemental also suffers more than most classes due to movement, though not nearly as badly as people seem to think nor as badly as we did in WotLK; you can check this by running simulationcraft with a "helter skelter" style fight instead of "patchwerk". The point of this post isn't to argue about Shaman DPS numbers or how I'm not factoring in movement, though, it is only to demonstrate the mathematical issues with relying on stateofdps.com data to discuss class balance. The numbers I'm quoting here are to prove a point in that regard, not to debate class balance in general.


On Sims vs. Logs

Since this point will inevitably come up at some point, a quick comment. The above factors, as well as human error, will ALWAYS be a factor to any kind of log analysis. A simulation, on the other hand, isn't affected by latency or lag, or human error, and can quickly punch up hundreds of hours of simulated combat. Logs have purposes; some classes do shine on certain fights, and you won't see that from a sim. But, in general, you only want to consider normal DPS and moving DPS for class balance comparisons, not "my class does amazingly well/badly on one gimmick fight". Using simulationcraft, you've got Patchwerk and Helter Skelter sims, for non movement and heavy movement, and the combination of the two should give you a good idea of how much movement is going to generally affect a given class.

Logs have purposes. They're the best way to improve your personal performance by checking uptimes and such. But they're not the best way to look at class balance. Sims are better for that purpose, since they eliminate the complicating factors the logs cannot, leaving class performance as the only remaining variable.

This post will be stickied, since I have had to post much the same in several threads and want a place to reference and link to. However, do NOT use this thread to discuss Shaman class balance. This thread is ONLY to be used to discuss the use of statistical methods of data comparison. I will simply delete anything that isn't relevant to that discussion, not because I am trying to censor the thread, but because I want to keep this sticky rigidly on a narrow topic. You're free to start a new topic and reference this thread, just don't do it here.

Again; this thread is NOT the place to discuss class balance or other such topics. Doing so will be treated, and infracted, as spam.

[Thomasaquina]

Great post. Another factor people can remember, which is probably a sub set of "player bias", is the gear gap between players. Because the scaling of dps based on gear is somewhat different by class, this can skew the data even more.

[Deleted]

Nicely explained, in clear understandable language.

[Iacom]

Funny that, because in A-level statistics we're doing the normal distribution right now.

[Bakis]

A much needed post. The two sites are useless in 99% of the cases simply cos the reader got no clue of what to make of it.

[StendhalSyndrom]

Damn you complex shammies!!!!!!


Edit: in light of all this new (clear) info I'm off to roll one now!

[Siph]

However, do NOT use this thread to discuss Shaman class balance.

Then get it stickied on a more general forum. If you stickied it here because your moderator rights are limited to the shaman forums you are just abusing your position, thousands of other posters also feel that their posts are more important than those of others.

OT: trying to lecture the masses on the internet with such a lengthy post is a wasted effort. Most of them will never get beyond looking at a 5-digit number on raid logs no matter what you tell them.

[Scuravolpe]

Thank you! Someone finally explains why statistics are not a reliable way of proving a mathematical argument. They can help sometimes, but are often horribly, horribly misused.

[reverendball]

and here i thought the fire mage standard deviation was higher than that

still a bit ridiculous, hopefully they implement more changes to make it more consistent

@OP good post, methinks a lot of ppl QQing will be redirected here so they can learn about maths and stats

[lifteez]

I think you hit the nail on the head on this topic. Well done OP

[Pet0r]

looks very interesting

[Endus]

Then get it stickied on a more general forum. If you stickied it here because your moderator rights are limited to the shaman forums you are just abusing your position, thousands of other posters also feel that their posts are more important than those of others.

OT: trying to lecture the masses on the internet with such a lengthy post is a wasted effort. Most of them will never get beyond looking at a 5-digit number on raid logs no matter what you tell them.

I was simply going to delete this, but I'm going to make a one-time exception to make this point.

I've stickied it here because I've posted the same information in multiple threads here, and it's a topic that comes up on the Shaman boards regularly. I also had other Shaman mods asking me to sticky it.

If I wanted it stickied in General, I could just poke any of the General mods I'm chatting with on IRC right now to do it for me. There's no abuse of power going on, it's solely that this is a topic that comes up regularly on these boards. Regularly enough that we wanted a sticky.

[Galith]

Thank you! Someone finally explains why statistics are not a reliable way of proving a mathematical argument. They can help sometimes, but are often horribly, horribly misused.

My thoughts exactly. I hate reading all of these statistics that people post that are intentionally misleading just to try and prove their own points.

[Radux]

Woot it's finally up!

One other thought on WoL specifically (top parses) and why they can't be accurate. People who rank so high on WoL are there specifically because they and their raid are helping them get spots. This can mean a Disc Priest feeding an Arcane Mage their PI's or Mages swapping their FM and potentially even Druid's giving them Innervates. It all adds up and completely skews data when people try to make a point with parses of this caliber.

As for luck factor. It's one thing to say there's lucky streaks in things like an Elemental Shaman getting an odd number of EO procs. It's another lucky thing to say that you have to move a minimal amount during an encounter because you aren't RNG chosen for something that requires you to stop DPS or DPS on the move, even.

[Endus]

Thank you! Someone finally explains why statistics are not a reliable way of proving a mathematical argument. They can help sometimes, but are often horribly, horribly misused.

My thoughts exactly. I hate reading all of these statistics that people post that are intentionally misleading just to try and prove their own points.

There's a reason for the saying; "There are lies, damn lies, and statistics". And that's not even a relatively recent statement; that's a comment by Mark Twain.

[tibben]

Another factor to consider is length of the encounter (overall guild ability)

Hypothetical Situation:
Let's say that you can sustain 40k dps for the duration of a bloodlust/heroism. Your buddy in another guild has the same gear, same skill level and same luck with crits/procs in his encounter as you do in yours. He also sustains 40k dps for the duration of his bloodlust/heroism. For the rest of the fight you both sustain an average of 25k dps.

Your fight lasts 5 minutes. His guild struggles a little, but brings it down in 6 minutes. That means you averaged (40k * .13333) + (25k * .86667) = 27k dps while your buddy averaged (40k * .11111) + (25k * .88889) = 26,667 dps. You both maximized your output and had the same luck with crits and procs, but because your guild can kill the boss faster you finished with an extra 333 dps.

Your guild killing a boss a full minute faster than another guild has very little to do with your individual skill as opposed to the collective skill of your guild as well as the other factors like if all your dps happen to get lucky proc streaks and crit streaks in the same encounter. Yet it can easily start to set you apart from other players of similar skill and gear. There's also the fact that the longer an encounter drags on the more likely you are to make a mistake in your rotation/priority system. We are just humans after all.

Whenever I try to compare myself to another player that was higher ranked on WoL than myself I always look for someone that had an encounter time within 2-3 seconds of my kill time as this should give me a better idea of if it was simply a matter of lucky crit streaks, extra buffs that I didn't have, or if he was simply doing something different with shot selections than I was.

[JackalopeSpam]

Just throwing this out there as another source of information that should be considered with the same cautions as mentioned above. However this one also allows from a small timeline to be added to the tracking at a lose of other bits of information. However it can be an interesting site to monitor around hot fixes.

http://raidbots.com/dpshistory/Overall/10N/#3vs00

[Anyasha]

Woot it's finally up!

One other thought on WoL specifically (top parses) and why they can't be accurate. People who rank so high on WoL are there specifically because they and their raid are helping them get spots. This can mean a Disc Priest feeding an Arcane Mage their PI's or Mages swapping their FM and potentially even Druid's giving them Innervates. It all adds up and completely skews data when people try to make a point with parses of this caliber.

This is speculative and, by and large, an urban myth. You can look at the majority of the top DPS on any given fight and check for PI, ToTT, Dark Intent, FM, Innervates, etc. Yes, there certainly *are* some people who 'cheat' by using those tactics. However, it is wholly unfair to say that the people in the top parses all use them. I have top10 lock rankings myself and was like 30sec from #1 all classes hMagmaw (he tantrumed....). I was not utilizing any cheap buff mechanics. The main culprit for 'cheap mechanic' usage are fights that favor certain classes by a huge margin.

People don't cheat their parses as much as other people force themselves to believe in order to justify why they cannot get the same numbers. This is like claiming every single Olympic athlete uses steroids because you and the majority of the population cannot be competitive with them.

[xebtria]

very nicely done. will use this thread in our guild... at least in the "spread the word"-style.

oh and there are some further sayings regarding statistics.

the first I just catched somewhere in some english speaking forum, maybe even here:
84,7% of all statistics are made up.

and the second is an - at least of my knowledge - german one, I hope I translate it correctly:
Never trust a statistic you didn't fake yourself.

I guess, those aren't very recent either, and yet they represent the very situation of the "abuse" of stateofdps and worldoflogs.

[Radux]

This is speculative and, by and large, an urban myth. You can look at the majority of the top DPS on any given fight and check for PI, ToTT, Dark Intent, FM, Innervates, etc. Yes, there certainly *are* some people who 'cheat' by using those tactics. However, it is wholly unfair to say that the people in the top parses all use them. I have top10 lock rankings myself and was like 30sec from #1 all classes hMagmaw (he tantrumed....). I was not utilizing any cheap buff mechanics. The main culprit for 'cheap mechanic' usage are fights that favor certain classes by a huge margin.

People don't cheat their parses as much as other people force themselves to believe in order to justify why they cannot get the same numbers. This is like claiming every single Olympic athlete uses steroids because you and the majority of the population cannot be competitive with them.

That's a fair judgment. My bad on jumping the gun a little with the comment. However, as time goes on (my god look at ICC), people get bored with the content and try to utilize their time wisely with high dps and attempts at WoL spots. I know my guild started doing it the lockout after we dropped H LK.

I wasn't trying to imply that it didn't take skill or whatever to get up there. It does. My comment may only pertain to a small variance, but unfortunately it's still something that needs to be looked at and whatnot when trying to comb through data.

And to be fair, not everyone really attempts to place on logs. I personally just use WoL as a tool to see where I screwed up so that I won't screw up again. Like many people, I want to utilize my raiding time with my guild wisely. If we can clear content in 3-4 hours in 2 days instead of 3? Yeah I'm going to put in the extra time combing through logs of wipes to ensure that the screw ups aren't my fault.