Nighthold DPS breakdown: redefining the debate on class balance (with data)

[Deleted]

Abstract

Hello everyone,

While this might be a bit of a long thread, I hope many of you will find this information useful. The post was motivated by previous discussions on balance that I followed, all of them suffering from cherry-picked percentile by fight rankings. This data is publicly available on warcraftlogs.com for everyone to make their own detailed assessments, just not (yet?) in this particular form. To spare you a novel, I won't interpret every fight individually and instead let the data speak for itself. My main conclusions, however, are the following:

1) It's no longer about balancing specs. The DPS rankings between fights change so drastically that flat adjustments are becoming a problem rather than a solution. With the disappearance of consistent underperformance, future discussions of balance should focus on the interactions between specific spells and specific fight mechanics.

2) There are currently very few combinations of spec and fight that seem to be outside of sensible margins. No spec stands out across all fights in either direction.

3) The variance within specs is much greater than between specs. Highly skilled players, under favourable circumstances, can push even the weakest specs into top DPS ranges.


To identify the mechanics that should be adjusted, this data should be complemented with a DPS breakdown by fight by spec by spell by percentile. That's a bunch of hyperplanes and would probably look very cool, but I don't want to take someone's job away here.

However, seeing all this I'm now a lot more inclined to buy into the official assessment of current class balance. It's also fascinating to see how much more the best players are able to get out of the different specs (notice the tighter distributions on Mythic) and which classes are the hardest to play. And it certainly shows that our friends at Blizzard are doing an impossible job quite well.

I'm looking forward to hearing your thoughts on this. Please do not hesitate to point out any mistakes that I might have made. This thread will be edited accordingly.


Thanks,

Katsu.


Update: You can find an overview by class here.

Data

All data used here was extracted from warcraftlogs.com on March 14th 2017. The settings were identical and included:

- Player Damage Statistics For
- Each boss
- Heroic (and Mythic)
- For all DPS
- For all item Levels, and for ilvl 899-901
- By percentiles, p10, p25, p50, p75, p90, max
- Over A Range Of 2 Weeks
- Current Standings

Regarding the source of the data, there are three main caveats:

1) Warcraftlogs is potentially biased towards including better players because many of those who are not playing competitively are also not uploading logs. They might be partially covered by other people's logs, but it's convincing to assume that the distributions should be somewhat shifted to the right. However, as long as this bias is consistent across all specs (essentially assuming that the spec distribution is the same for covered and non-covered players) the shift is constant and thus of no direct importance to interpretation.
2) There's also the issue that many of the best logs are private during times of progress, thus blurring the picture at the very top end of things. This could potentially move the outliers a bit, but if the same assumtion holds as in 1) it can also be ignored. It becomes meaningful if the discussion is to be had what to balance for.
3) Data on Mythic is scarce for some fights, so interpretations should be taken with a grain of salt.

The data is prepared as boxplots, with a few alterations. The black whiskers correspond the the first and the ninth decile. In this case, the left minimum is essentially zero (someone died without doing any damage). The right red whisker corresponds to the maximum recorded DPS value. The median value is indicated as p50 and the data is sorted accordingly. For each fight, data is provided for Heroic and Mythic difficulty and for all item levels as well as for Heroic difficulty at item level 899-901 (to eliminate the effect of gear scaling over the course of the examination period of two weeks).

---

DPS breakdown by fight

Skorpyron
Heroic


Mythic


Heroic 899-901



Anomaly
Heroic


Mythic


Heroic 899-901


Trilliax
Heroic


Mythic


Heroic 899-901



Spellblade
Heroic


Mythic


Heroic 899-901



Krosus
Heroic


Mythic


Heroic 899-901



Augur
Heroic


Mythic


Heroic 899-901



Tichondrius
Heroic


Mythic


Heroic 899-901



Botanist
Heroic


Mythic


Heroic 899-901



Elisande
Heroic


Mythic


Heroic 899-901



GulDan
Heroic


Mythic


Heroic 899-901

[IceMan1763]

It would be interesting to see how much the proper legendaries affect dps ranges.

[ufta]

Very interesting indeed.
First off, thank you for taking the time to put this together.
One thing this shows that I find interesting is that, on almost any given fight, you could be playing the worst-possible performing spec in the game FOR THAT ENCOUNTER, but if you perform in the 95th percentile, and everyone else in your raid performed at the 75th percentile.....you would top the meter.

[PraisCthulhu]

Very interesting indeed.
First off, thank you for taking the time to put this together.
One thing this shows that I find interesting is that, on almost any given fight, you could be playing the worst-possible performing spec in the game FOR THAT ENCOUNTER, but if you perform in the 95th percentile, and everyone else in your raid performed at the 75th percentile.....you would top the meter.

This is a sign that the game is actually in a pretty solid spot balance wise. Yes there will always be a best and worst performing spec, that is just the nature of the game. But, this shows that player skill accounts for more of your DPS than your class.

[Deleted]

Having personal experience with Destruction and Outlaw my impressions are borne out by the graphs. The mythic Outlaw numbers are driven by the fact that whoever is doing mythic as Outlaw has Greeenskins. These two specs are not rewarding players with fair performance and hence my warlock and rogue were moved to Affli and Assassination which are greatly more viable without any requirement on legendary.

We also see Arms and Survival as lagging behind. Not sure there's any hope for destro and outlaw in this expac.

[schwarzkopf]

There is also an additional bias in that good players will play certain specs over other specs magnifying what may have been small differences.

e.g. if Fire is actually 0.1% better than Arcane, top notch players will go Fire - meaning the measured difference will include an additional bias of better players playing Fire.

However, your charts seem to match up with where I sit on various fights as arcane .... within my own group.

[KAMBO1431]

I find it annoying how WCL uses a score which doesnt show what I look for, so this is great! Awesome stuff.

The variance between classes is so small for the Mythic fights. Amazing.

I can't figure out Q3, how have you represented the Outliers? Or what is the translation for an Outlier in WCL colour terms? Im assuming orange.

One request, can you please share the dataset? Or add more charts showing this information by Class (a lot of work hehe)?
e.g.
Warlock Heroic
Skorpyron <box plot>
Anomaly <box plot>
Trilliax <box plot>
etc

Mage Heroic
Skorpyron <box plot>
Anomaly <box plot>
Trilliax <box plot>
etc etc

[Deleted]

First of all, thank you all for your feedback. Let me address some of the points that were made one by one:

It would be interesting to see how much the proper legendaries affect dps ranges.

Indeed, but I'm not sure if this is currently feasible in a systematic fashion. I'll look into this some more and see what I can find. For now I'm inclined to think that their impact outside of the highest parses should be relatively small compared to the spread induced by player skill. Furthermore, at this point the argument that more people with the best legendaries should be among the highest ranking parses remains an unproven assumtion. There could be just as many bad players with the right legendaries at the bottom half of the distribution.

Personally I'm convinced that people use the argument of not having gotten the right legendaries as a subjective excuse for their own objectively poor performance. I've had plenty of raids where the people who complained the most turned out to rank way below the median when we evaluated the logs. As long as you're not ranking in the lower 90s you should not worry about the impact of legendaries and instead focus on becoming a better player. There are players who do not have the best legendaries among the top parses for every class.

This is a sign that the game is actually in a pretty solid spot balance wise. Yes there will always be a best and worst performing spec, that is just the nature of the game. But, this shows that player skill accounts for more of your DPS than your class.

Yes. If anything, this is the one thing everyone should take away from this thread.

Having personal experience with Destruction and Outlaw my impressions are borne out by the graphs. The mythic Outlaw numbers are driven by the fact that whoever is doing mythic as Outlaw has Greeenskins. These two specs are not rewarding players with fair performance and hence my warlock and rogue were moved to Affli and Assassination which are greatly more viable without any requirement on legendary.

We also see Arms and Survival as lagging behind. Not sure there's any hope for destro and outlaw in this expac.

Personal experience is a bad indicator for what’s actually going in the game on a meta level. You make a few statements without backing them up, which is why I think they should be taken with a grain of salt.

First of all, how do you know that “whoever is doing mythic as Outlaw has Greenskins”? The data doesn’t show that and I’m not aware of any data that does (without it being a massive struggle to get the information). It’s certainly not a fact. There might very well be players in the Mythic pool that do well without it. As I mentioned before, I’m trying to see if there’s a solid way to assess this (I’m quite curious about this myself) and will get back to you as soon as I find something.

Secondly, you state that these specs “are not rewarding players with fair performance”. This is false. If played well, Destruction can compete for top ranks (as can any spec). Destruction is also by far the highest performing spec on Elisande Mythic, a fight that’s still in the realm of progress for many guilds. Outlaw has a strong inherent RNG element, I agree with you, but so do many other specs (Arms is essentially a moving slot machine). It does reward good performance just as well as the other specs do, just less consistently.


I disagree with your assesment of the classes, but on a more fundamental level. This kind of thinking is outdated, that's the point of this thread. Please do not generalize things, that's no longer appropriate.

There is also an additional bias in that good players will play certain specs over other specs magnifying what may have been small differences.

e.g. if Fire is actually 0.1% better than Arcane, top notch players will go Fire - meaning the measured difference will include an additional bias of better players playing Fire.

However, your charts seem to match up with where I sit on various fights as arcane .... within my own group.

You’re making an assumption here that requires several others. First of all, you assume that good players will switch as soon as one spec is “better”. As I’ve shown above, you first would have to define what “better” actually means. All specs have overlapping DPS distributions for all fights, so agreeing on this definition is already tricky.

You’re also assuming that someone who is able to perform one spec at p99 is able to do the same thing on another. That’s a tall order. Even if you’re able to do that, the random component of DPS at the very top level is so large that your switch could still turn out not to be worthwhile.

Regardless, it’s an interesting question whether or not this behavior is actually happening. So I looked into it. I’ll add the results of a new analysis in another post. The conclusion is that it does, but the effect of switching has to be disentangled from the effect of stacking. Both happens, but the effects differ by class and spec.

In the end, the distributions might indeed be somewhat affected as you say, but since the data is still coming in for the most difficult fights I would not draw any conclusions from it.

I find it annoying how WCL uses a score which doesnt show what I look for, so this is great! Awesome stuff.

The variance between classes is so small for the Mythic fights. Amazing.

I can't figure out Q3, how have you represented the Outliers? Or what is the translation for an Outlier in WCL colour terms? Im assuming orange.

One request, can you please share the dataset? Or add more charts showing this information by Class (a lot of work hehe)?
e.g.
Warlock Heroic
Skorpyron <box plot>
Anomaly <box plot>
Trilliax <box plot>
etc

Mage Heroic
Skorpyron <box plot>
Anomaly <box plot>
Trilliax <box plot>
etc etc

Q2 is grey, Q3 is orange, the black whisker is Q3-p90 and the red whisker is p90-max. On the left side the minimum is zero by definition, so I didn’t show anything below p10.

I’ve already started to do that for Mages over at altered time. I’ll add the information in another post.

[KAMBO1431]

First of all, thank you all for your feedback. Let me address some of the points that were made one by one:



Indeed, but I'm not sure if this is currently feasible in a systematic fashion. I'll look into this some more and see what I can find. For now I'm inclined to think that their impact outside of the highest parses should be relatively small compared to the spread induced by player skill. Furthermore, at this point the argument that more people with the best legendaries should be among the highest ranking parses remains an unproven assumtion. There could be just as many bad players with the right legendaries at the bottom half of the distribution.

Personally I'm convinced that people use the argument of not having gotten the right legendaries as a subjective excuse for their own objectively poor performance. I've had plenty of raids where the people who complained the most turned out to rank way below the median when we evaluated the logs. As long as you're not ranking in the lower 90s you should not worry about the impact of legendaries and instead focus on becoming a better player. There are players who do not have the best legendaries among the top parses for every class.




Yes. If anything, this is the one thing everyone should take away from this thread.




Personal experience is a bad indicator for what’s actually going in the game on a meta level. You make a few statements without backing them up, which is why I think they should be taken with a grain of salt.

First of all, how do you know that “whoever is doing mythic as Outlaw has Greenskins”? The data doesn’t show that and I’m not aware of any data that does (without it being a massive struggle to get the information). It’s certainly not a fact. There might very well be players in the Mythic pool that do well without it. As I mentioned before, I’m trying to see if there’s a solid way to assess this (I’m quite curious about this myself) and will get back to you as soon as I find something.

Secondly, you state that these specs “are not rewarding players with fair performance”. This is false. If played well, Destruction can compete for top ranks (as can any spec). Destruction is also by far the highest performing spec on Elisande Mythic, a fight that’s still in the realm of progress for many guilds. Outlaw has a strong inherent RNG element, I agree with you, but so do many other specs (Arms is essentially a moving slot machine). It does reward good performance just as well as the other specs do, just less consistently.


I disagree with your assesment of the classes, but on a more fundamental level. This kind of thinking is outdated, that's the point of this thread. Please do not generalize things, that's no longer appropriate.



You’re making an assumption here that requires several others. First of all, you assume that good players will switch as soon as one spec is “better”. As I’ve shown above, you first would have to define what “better” actually means. All specs have overlapping DPS distributions for all fights, so agreeing on this definition is already tricky.

You’re also assuming that someone who is able to perform one spec at p99 is able to do the same thing on another. That’s a tall order. Even if you’re able to do that, the random component of DPS at the very top level is so large that your switch could still turn out not to be worthwhile.

Regardless, it’s an interesting question whether or not this behavior is actually happening. So I looked into it. I’ll add the results of a new analysis in another post. The conclusion is that it does, but the effect of switching has to be disentangled from the effect of stacking. Both happens, but the effects differ by class and spec.

In the end, the distributions might indeed be somewhat affected as you say, but since the data is still coming in for the most difficult fights I would not draw any conclusions from it.




Q2 is grey, Q3 is orange, the black whisker is Q3-p90 and the red whisker is p90-max. On the left side the minimum is zero by definition, so I didn’t show anything below p10.

I’ve already started to do that for Mages over at altered time. I’ll add the information in another post.

I'm also a fire mage and just saw your post on AT, scrap my request for a box plot by boss for classes, charts like the 'Mage DPS by percentile ilvl 895 - Augur Heroic' tell me what I want to know. If you ever get around to putting all the bosses up that would be fantastic.

Keep up the good work. I've never really been motivated to play frost but if I can make something easier for myself and keep the output the same or better, why not hey.

I'll always have Skorpyron/Tich to go into as fire

[Deleted]

I like that you added i level this is core imo.
As a feral i am still not satisfied because there is not a single fight we perform top 5 or even 6? While having a complex rota

[Deleted]

Very interesting indeed.
First off, thank you for taking the time to put this together.
One thing this shows that I find interesting is that, on almost any given fight, you could be playing the worst-possible performing spec in the game FOR THAT ENCOUNTER, but if you perform in the 95th percentile, and everyone else in your raid performed at the 75th percentile.....you would top the meter.

This is a logical fallacy for the high end of performance this expansion. At a certain level of player performance (for the sake of it lets say 95% of the optimum), player skill becomes irrelevant and legendaries become the defining factor. When you progress mythic a a decent level, it doesn't help you a lick that you are able to (going full cliché) outperform a 50-year old housewife even if she has 30 item levels more than you. That is irrelevant for balance discussions which should be based on equal player skill and gear.

Even if I reach my simulated 720k DPS for the Krosus mythic fight (which I did or one of my kills with good RNG a few weeks back) it does not even put me at the 70th percentile. Why? Because I have only one of the 3 best DPS legendaries. You can look up my character, my gear is pretty good. Since I analyze my logs in depth, I can guarantee you that I play my rotation cleaner than almost any other player and as close to "perfect" as you can be. But you can't overcome legendaries making up 10s of thousands of DPS.

It is not a sign of good balance that the absolute best Balance druids with BiS gear and great RNG can beat braindead Demon Hunters or whatever. It just isn't.

btw: saying someone plays their class to 95% (or whatever) of their potential is not equal to them being better than 94% of the other players. Utilization your classes potential is not a linear graph since even bad players do at least 30-40% of the potential DPS. So there are many, many players that are doing pretty good on that front.

[Niwes]

i dont get why ppl thinking EVERY dps discussion would be still useful at this point?

as OP said, specs dps are more close together than they were ever before. the gap between the horrible and the top spec is smaller than ever (and thats a good thing, applause to blizzard).

but at the same time, 2 slots have that immense effect (legendaries) that everything else is uninteressting. even when the gaps betweeen the specs would be that huge, like they were in previous tiers, the legendaries would change everything. but with this small gaps its even more worse. its like you put a magnifying class on it.

an example:

you can give a not so skilled frost mage player (or whatever the worst spec is) 2 BiS legendaries, 1 or 2 good trinkets and good relics, and he outperforms every high skilled assa rogue player (or whatever the top spec is) with average version of said items, by a far margin. without any problems.

in a world like this, literally EVERY discussion about specs dps balance is totally useless, unless they take the said items into account.

and at this point, we dont have even talked about the absurd heavyness of RNG in spell rotations! both things together (items and RNG behaviour) make any usefull comparrisson totally irelevant.

and there is, last but not least, the impactful "grind for your life" difference (i.e. HC raider ppl that grind their ass of and raid with 20 points in last trait, doing ~8% more dmg than raider who let AP slowly come in by just raiding). (ofc this is moe a problem in HC than in Myth bc Myth ppl all have the last 20 points).

i for myself stopped every discussion about dps balance when i weekly raid my enh shaman in HC guild runs and perform shitty as hell, with 29-46 on warcraftlogs, on some fights, while our warrior performs 77-99, and i just be literally 3-7% behind him (and be the next or overnext on the dmg list in the logs). thats just laughable.

as long as we are in Legion, with Legendary RNG, relics RNG, trinkets RNG, spell rotation RNG and a so much AP impact, EVERY dps discussion is totally useless.

[Ifeanychukwu]

Wish they'd buff boomkins

[rda]

1) It's no longer about balancing specs. The DPS rankings between fights change so drastically that flat adjustments are becoming a problem rather than a solution. With the disappearance of consistent underperformance, future discussions of balance should focus on the interactions between specific spells and specific fight mechanics.

2) There are currently very few combinations of spec and fight that seem to be outside of sensible margins. No spec stands out across all fights in either direction.

3) The variance within specs is much greater than between specs. Highly skilled players can push even the weakest specs into top DPS ranges.

Once you start talking about "expected average performance" by reducing the logs to those between 25th and 75th percentile - shown in your graphs as gray/yellow bars - I don't see (3) and, on half of the bosses, don't see (2) either.

Frankly, all your analysis is based on taking into account huge shoulders from 90th percentile to 100th. The shoulders are this huge precisely because there are big outliers, yet your analysis somehow concludes from the size of shoulders that the balance is kind of fine - because "the variance within specs (read: the presence of big outliers) is much greater than between specs". This is a bogus conclusion.

The numbers are great, thanks for collecting them, the charts look nice, but the analysis is garbage.

- - - Updated - - -

This is a sign that the game is actually in a pretty solid spot balance wise. Yes there will always be a best and worst performing spec, that is just the nature of the game. But, this shows that player skill accounts for more of your DPS than your class.

This is not at all the case, see above.

The conclusion of the variance being bigger within specs than between them exists almost entirely due to the presence of big shoulders between 90th and 100th percentile. These shoulders happen not because of player skill but rather because of unusual circumstances (things proccing at a very opportune moment, the entire raid trying to beat the encounter by propping the damage of a particular player or player group at the cost of others, etc).

The balance shown in the charts isn't great.

If we stick to talking about average expected performance by only looking into the gray/yellow bars, on the very first boss half of the specs can not reach the mid point of the top performer. You roll a wrong class, you can't beat the top. On the second boss, everyone in the bottom spec is outperformed by everyone in the top spec. Etc. That's called "the balance is bad".

[Deleted]

Once you start talking about "expected average performance" by reducing the logs to those between 25th and 75th percentile - shown in your graphs as gray/yellow bars - I don't see (3) and, on half of the bosses, don't see (2) either.

Frankly, all your analysis is based on taking into account huge shoulders from 90th percentile to 100th. The shoulders are this huge precisely because there are big outliers, yet your analysis somehow concludes from the size of shoulders that the balance is kind of fine - because "the variance within specs (read: the presence of big outliers) is much greater than between specs". This is a bogus conclusion.

The numbers are great, thanks for collecting them, the charts look nice, but the analysis is garbage.

- - - Updated - - -

This is not at all the case, see above.

The conclusion of the variance being bigger within specs than between them exists almost entirely due to the presence of big shoulders between 90th and 100th percentile. These shoulders happen not because of player skill but rather because of unusual circumstances (things proccing at a very opportune moment, the entire raid trying to beat the encounter by propping the damage of a particular player or player group at the cost of others, etc).

The balance shown in the charts isn't great.

If we stick to talking about average expected performance by only looking into the gray/yellow bars, on the very first boss half of the specs can not reach the mid point of the top performer. You roll a wrong class, you can't beat the top. On the second boss, everyone in the bottom spec is outperformed by everyone in the top spec. Etc. That's called "the balance is bad".

Great points, very well written.

To say the balance is as good as it has ever been between all specs when the bottom 25% (including many, many (early) death-logs) of one spec outperform 75% of another on nearly all fights is ridiculous =)

[Deleted]

Once you start talking about "expected average performance" by reducing the logs to those between 25th and 75th percentile - shown in your graphs as gray/yellow bars - I don't see (3) and, on half of the bosses, don't see (2) either.

Frankly, all your analysis is based on taking into account huge shoulders from 90th percentile to 100th. The shoulders are this huge precisely because there are big outliers, yet your analysis somehow concludes from the size of shoulders that the balance is kind of fine - because "the variance within specs (read: the presence of big outliers) is much greater than between specs". This is a bogus conclusion.

The numbers are great, thanks for collecting them, the charts look nice, but the analysis is garbage.

- - - Updated - - -



This is not at all the case, see above.

The conclusion of the variance being bigger within specs than between them exists almost entirely due to the presence of big shoulders between 90th and 100th percentile. These shoulders happen not because of player skill but rather because of unusual circumstances (things proccing at a very opportune moment, the entire raid trying to beat the encounter by propping the damage of a particular player or player group at the cost of others, etc).

The balance shown in the charts isn't great.

If we stick to talking about average expected performance by only looking into the gray/yellow bars, on the very first boss half of the specs can not reach the mid point of the top performer. You roll a wrong class, you can't beat the top. On the second boss, everyone in the bottom spec is outperformed by everyone in the top spec. Etc. That's called "the balance is bad".

It is just as bogus basing the analysis of the most common cases (25th-75th percentile) because you lose the cases where people are highly skilled as they aren't common. There is indeed also a huge amount of outliers that skew the analysis as you correctly propose. But you cannot just disregard all high percentiles because some of them are due to extraordinary circumstances.

If you base your analysis on the most common case, you will see the specs with the easiest rotation being higher on the rankings than they should be. Simply because you have removed those that manage to play difficult (and assuming rewarding) rotations better, because they consistently log over the 75th percentile for their spec.
And then you simply disregard 3) because of selection bias, and that is just as bad as not adressing outliers.

[Deleted]

It is just as bogus basing the analysis of the most common cases (25th-75th percentile) because you lose the cases where people are highly skilled as they aren't common. There is indeed also a huge amount of outliers that skew the analysis as you correctly propose. But you cannot just disregard all high percentiles because some of them are due to extraordinary circumstances.

If you base your analysis on the most common case, you will see the specs with the easiest rotation being higher on the rankings than they should be. Simply because you have removed those that manage to play difficult (and assuming rewarding) rotations better, because they consistently log over the 75th percentile for their spec.
And then you simply disregard 3) because of selection bias, and that is just as bad as not adressing outliers.

You missed his point: as long as legendaries and RNG (for some specs) make up 10+% DPS differences, it makes zero sense to include top logs.

My DPS as a Havoc DH (high variance due to the high bursts) on mythic Krosus literally vary by 60k DPS (730k-670k) from one try to the next. Give me one of the 2 legendaries I still want and that becomes 100+k DPS regardless of player skill.

Before we had this shit legendary system, you could reasonably take 90th or even 95th percentile logs, because there were enough of them and the top parses were close enough in terms of gear that it made sense. When having the right legendaries means the difference between doing 650k vs. 800+k, all balance discussion become pointless.

[Deleted]

You missed his point: as long as legendaries and RNG (for some specs) make up 10+% DPS differences, it makes zero sense to include top logs.

My DPS as a Havoc DH (high variance due to the high bursts) on mythic Krosus literally vary by 60k DPS (730k-670k) from one try to the next. Give me one of the 2 legendaries I still want and that becomes 100+k DPS regardless of player skill.

Before we had this shit legendary system, you could reasonably take 90th or even 95th percentile logs, because there were enough of them and the top parses were close enough in terms of gear that it made sense. When having the right legendaries means the difference between doing 650k vs. 800+k, all balance discussion become pointless.

When was that ever the point? None of those post I refer to had anything to do with legendaries. And even if that was the point, the sensible idea would be to limit the dataset to only those having BiS legendaries, not excluding them when analyzing inter-class/spec damage in the tails of dps (I.E. to account for balance between highly skilled players), and make an analysis of that subset of the sample. To disregard BiS legendaries simply means you make the comparison based on the middle-of-the-road legendaries mixed with BiS legendaries obtained by worse players. That does not serve for a better analysis of spec balance.

And you are making the exact same mistake, as I argued. Some of the high logs are due to extraordinary circumstances. That is a problem indeed. To cast away those that are not due to extraordinary circumstances (and no BiS legendaries are not extraordinary circumstances) is also a problem (possibly even a bigger problem, especially as more parses are on record). Which is why neither of the two conclusions can stand alone - in best case they are complementary.

[Raiju]

Give me one of the 2 legendaries I still want and that becomes 100+k DPS regardless of player skill.

Ring got nerfed a long time ago, it's bis but its going to be a small increase over your current legendaries.

[rda]

It is just as bogus basing the analysis of the most common cases (25th-75th percentile) because you lose the cases where people are highly skilled as they aren't common. There is indeed also a huge amount of outliers that skew the analysis as you correctly propose. But you cannot just disregard all high percentiles because some of them are due to extraordinary circumstances.

If you base your analysis on the most common case, you will see the specs with the easiest rotation being higher on the rankings than they should be. Simply because you have removed those that manage to play difficult (and assuming rewarding) rotations better, because they consistently log over the 75th percentile for their spec.
And then you simply disregard 3) because of selection bias, and that is just as bad as not adressing outliers.

It's not "just as bogus".

We have no good way of distinguishing high parses that happened due to players being skilled and high parses that happened due to random procs / artificially created special circumstances. If we had a good way to do that, we would have filtered out all of the latter and then we could proceed to analyzing what's left. It would still make sense to analyze the middle and the ends (both high and low) separately, but at least the ends would be analyzed.

But since we have no good way to filter out all of the fluff and the history shows that quite a lot of outliers indeed happen because of random procs and things like that - take 2-3 top graphs for any boss and I bet at least one of them - quite possibly all of them - will contain things like that, we can either filter heavily and try to analyze the averages, speaking about the results as those for averages, or we have to forfeit the analysis completely because we already know the outliers are skewing it heavily and tons of them are artificial.