The Trouble with Tier lists...

[Malkiah]

What you're describing here is selection bias. You're absolutely right in what you're saying - the top performers don't represent the average player. Analysing just the top 5% of players doesn't reflect what you're going to see in game outside most of the time. Obviously the higher level of content you're doing the more likely you are to run into players in that top 5%, but if you're grouping with them regularly chances are that you're also in that top 5%.

yeah like for example... M+ tier lists are generated by taking the top 100 recorded timed only runs and then averaging out the distribution of specs within those runs and then giving a grade based on representation of the spec.
it's literally just "how many of X spec was present in the top 100 runs" - it's absolutely NOTHING to do with performance in any way, because there is currently no way to measure performance in M+ runs.
(well, obviously there's something to be said for inference, but even that is so incredibly biased and faulty data in this context it's not terribly informative)
not only that, but M+ tier lists don't even look at keys under 16, so most of the "data" is coming from 22-24 keys.

basing your opinion of spec performance on 'tier' lists is like having 5 gallons worth of liquid you need to store and looking at 50, 75, and 100 gallon containers and nit-picking over which one is best.

To sumarise, a Tier list is an opinion of what the best classes are. Nothing more, nothing less.

and hell that's not even accurate.
M+ tier lists is just a graph showing how many of each spec show up in the top runs for the highest logged key runs.
those aren't "the best classes" in any realistic use of the term - those are what the top end players perceive to be the best classes for the way that they run keys... perhaps interesting data to have if you're idly curious about the fractional end of the delta, but other than as a curio it's utterly useless information with no relation to any material condition existing inside the game for 99% of players.

- - - Updated - - -

There is no "trouble" with tier lists. The trouble is how this community interprets tier lists. Tier lists on their own are just that: Easy ways to summarize mountains of information into a nice, clean visual which compartmentalizes everything in a way that, at a glance, you know broadly where something lands on the spectrum of good to bad.

but again though... no they don't even do that.

maybe an argument could be made that dps tier lists for raiding can give you a rough estimation of how good the throughput is for a given class in a raid environment, but the M+ tier lists are such selectively niche data that they are effectively utterly useless and tell you nothing of value.

"what car can reach the highest top speed in 30 seconds on a perfectly flat surface in ideal weather conditions when operated by a professional race driver who's only goal is to get as fast as possible" isn't useful information for someone deciding which car to buy to commute to work in a large urban city.

[huth]

Did you REALLY just claim to have a better understanding of the definition of a word than.....the Collins fucking dictionary?

At some point in your life, you need to accept "maybe I don't know everything, maybe I'm wrong about this". I think that point might be when you start arguing with the dictionary about the definition of a word.

Statistics are facts. If it's not a fact, it's not a statistic, by definition.

Long story short? Yes. Those guys have no clue what they're talking about and that isn't the correct definition of the term. Which isn't surprising, it's a dictionary, not a mathematical treatise. It's written by people who are not familiar with the subject. Dictionaries are not some infallible source of wisdom.

And no, statistics aren't facts. An average over a random assortment of values is a statistic. It's not a fact, however. Just a number.

[Femininity]

Feelycraft strikes again!

[Caprias]

Even though the data can never be perfect and always needs context, subcreation is doing a pretty neat job.

Edit: *if you're running that content that's listed (20-24 keys), you won't have as great of a time as destro/frost mage/balance druid/shadow below 20 keys for example.

[Relapses]

yeah like for example... M+ tier lists are generated by taking the top 100 recorded timed only runs and then averaging out the distribution of specs within those runs and then giving a grade based on representation of the spec.
it's literally just "how many of X spec was present in the top 100 runs" - it's absolutely NOTHING to do with performance in any way, because there is currently no way to measure performance in M+ runs.
(well, obviously there's something to be said for inference, but even that is so incredibly biased and faulty data in this context it's not terribly informative)
not only that, but M+ tier lists don't even look at keys under 16, so most of the "data" is coming from 22-24 keys.

basing your opinion of spec performance on 'tier' lists is like having 5 gallons worth of liquid you need to store and looking at 50, 75, and 100 gallon containers and nit-picking over which one is best.


and hell that's not even accurate.
M+ tier lists is just a graph showing how many of each spec show up in the top runs for the highest logged key runs.
those aren't "the best classes" in any realistic use of the term - those are what the top end players perceive to be the best classes for the way that they run keys... perhaps interesting data to have if you're idly curious about the fractional end of the delta, but other than as a curio it's utterly useless information with no relation to any material condition existing inside the game for 99% of players.

- - - Updated - - -


but again though... no they don't even do that.

maybe an argument could be made that dps tier lists for raiding can give you a rough estimation of how good the throughput is for a given class in a raid environment, but the M+ tier lists are such selectively niche data that they are effectively utterly useless and tell you nothing of value.

"what car can reach the highest top speed in 30 seconds on a perfectly flat surface in ideal weather conditions when operated by a professional race driver who's only goal is to get as fast as possible" isn't useful information for someone deciding which car to buy to commute to work in a large urban city.

I sold cars at one point and almost every single person I sold a car to asked about 0-60 times... for fucking Toyota Camrys.

[Woobels]

As someone who has been into fighting games for over a decade tier lists have largely become a meme for content, though the WoW community takes them too seriously.

[Relapses]

As someone who has been into fighting games for over a decade tier lists have largely become a meme for content, though the WoW community takes them too seriously.

I'd wager tier lists have always existed in WoW, we just didn't collate the information in them like we do these days. Before tier lists, people would just look at the WCL DPS charts and create their own headcanon. Before WCL, it was World of Logs. Before WoL, it was... uh... forums? lol. At any point, however, the game has always neatly appropriated good and bad and a meta has emerged. One could argue the meta of years gone was less regimented (and I'd 100% agree after seeing how WoW Classic played out), but it's always been there.

[Metacrias]

I taught statistics for a couple years - hated it. Reading through this thread's discussion of 'what is statistics' is going to give me nightmares.

[arkanon]

Long story short? Yes. Those guys have no clue what they're talking about and that isn't the correct definition of the term. Which isn't surprising, it's a dictionary, not a mathematical treatise. It's written by people who are not familiar with the subject. Dictionaries are not some infallible source of wisdom.

And no, statistics aren't facts. An average over a random assortment of values is a statistic. It's not a fact, however. Just a number.

Wait till you find out an average is a fact. I really think you should open up a dictionary some time, it's gonna blow your mind. Statistics are facts, and that is.........a fact.

[Gestopft]

Back in the day when Tier lists were just becomming a thing the people putting them out used actual anecdotal evidence and real game testing and comparisons to make them for each class. Later several top performing players of each class and spec would contribute to form a sort of a meta aggregate Tier list for more accurate data because lets face it not EVERYONE is going to be a top performer for each spec.

Nowadays however what I have come to realize is that Tier lists are mostly inaccurate and just thrown together with sim data and theory crafting.

I mean, when the only available data is based on PTR or beta, you're dealing with a small player base and a very limited data set. Any reasonable purveyor of tier lists is going to be continually updating the list based on available information, because things DO change; sometimes it's buffs/nerfs, sometimes it's players figuring out things that weren't apparent yet.

I mean, for this first season of DF, what about the tier lists was egregiously wrong before nerfs/buffs happened? Most that I saw put MM below where it should've been, because the burst is actually really good....and Fury Warriors started really strong but fell off hard...what has the 'aggregate' of the predictorate badly mangled?

- - - Updated - - -

Yeah I agree. Right now I'm playing Survival hunter which is what got me on this topic. For one I'm doing very well numbers wise. For two when I actually looked at the supposed statistical analysis of a couple of these tier-lists I noticed that the actual number of survival hunters in which they based their data on was fairly low.

For SV in particular- while you can do well if you are a good player of Survival, part of their low rating is that a) if you have a skilled player of all three hunter specs, the MM will have MUCH more burst, and the BM will probably have better sustain, both while being able to function at range, and b) hunter survivability is utterly awful, and that actually really matters this season, especially compared to previous seasons. I like to call hunters the "I took the least damage but somehow still died the most" class.

[Malkiah]

I'd wager tier lists have always existed in WoW, we just didn't collate the information in them like we do these days. Before tier lists, people would just look at the WCL DPS charts and create their own headcanon. Before WCL, it was World of Logs. Before WoL, it was... uh... forums? lol. At any point, however, the game has always neatly appropriated good and bad and a meta has emerged. One could argue the meta of years gone was less regimented (and I'd 100% agree after seeing how WoW Classic played out), but it's always been there.

it's an interesting point of divergence - because historically, it was just raid logging. that feels like it's at least in the area of data that can be extrapolated into sets of information that roughly equate to something actually happening in the game.

if your goal is to see which classes can do the most damage or healing, taking 50,000 logs of the same fight from a random sample of the community can give you a pretty reasonable idea of what is the average damage output of each class.
sadly, if your goal is to see which class can do the most damage or healing in M+, taking a few thousand completion results out of the 0.1% top end of a distribution range and then averaging out what specs show up the most, that is not telling you fuck-all.

i guess... what are we even talking about with 'tier lists'?
because raid log samples have been in the game for ages, you're right. but M+ rankings are both new, and currently based on zero practical data.

why don't we M+ log?
i'm no coder, of websites or of mathematical models. would it really be such a challenge to make an equivalent of wowlogs for M+?

[Relapses]

why don't we M+ log?
i'm no coder, of websites or of mathematical models. would it really be such a challenge to make an equivalent of wowlogs for M+?

M+ is logged on WCL, there simply aren't a lot of people uploading the logs (at least comparatively to raids). I'd wager the main reason is that each M+ dungeon is like an extremely long boss encounter so there isn't as much of a push to parse like in raids. You either have the damage, mechanics and coordination to time a difficult key or you don't. There isn't a whole lot of grey area.

[Ryzeth]

why don't we M+ log?
i'm no coder, of websites or of mathematical models. would it really be such a challenge to make an equivalent of wowlogs for M+?

Because most people running high keys don't care about parsing in them, so there's no real point to log. If nobody's uploading logs for them, then you're... Not gonna see logs for them... It's not as if the game automatically uploads fights to warcraftlogs, it's entirely opt-in and the website already has support for m+ logs as is.



The problem with tier lists have always been that most of them don't really take into account group utility. An enhance Shaman being 8th on a tier list isn't gonna "matter" when their windfury totem is giving the rogues/warriors/dks etc so much more dps as well, that logs don't show. Same can be said for something like a Shadow Priest... Logs don't reflect the increased dps of whoever they're giving PI to.

[Free]

I mean, when the only available data is based on PTR or beta, you're dealing with a small player base and a very limited data set. Any reasonable purveyor of tier lists is going to be continually updating the list based on available information, because things DO change; sometimes it's buffs/nerfs, sometimes it's players figuring out things that weren't apparent yet.

I mean, for this first season of DF, what about the tier lists was egregiously wrong before nerfs/buffs happened? Most that I saw put MM below where it should've been, because the burst is actually really good....and Fury Warriors started really strong but fell off hard...what has the 'aggregate' of the predictorate badly mangled?

- - - Updated - - -



For SV in particular- while you can do well if you are a good player of Survival, part of their low rating is that a) if you have a skilled player of all three hunter specs, the MM will have MUCH more burst, and the BM will probably have better sustain, both while being able to function at range, and b) hunter survivability is utterly awful, and that actually really matters this season, especially compared to previous seasons. I like to call hunters the "I took the least damage but somehow still died the most" class.

MM has been buffed 3 times now
not the best example lol

[Biomega]

why don't we M+ log?

Several reasons.

Routing, timing, etc. introduce so many variables that it's almost impossible to find meaningful comparisons that logs would be useful for. They're okay for some broad, general questions - but very few people are interested in those.

Key levels also make it so that there's a gazillion categories you'd need to find log comparisons. M+1 to M+27/28/29/however high it'll go. That's almost 30 key levels alone that you'd need to compare across. And then there's affixes. Makes little sense to compare Tyrannical to Fortified, for starters. So double the number of keys again. Want healing logs? Huge difference if it's, say, Grievous week. Even more. And so on and so forth. You'd end up with a matrix so convoluted no one could set it up right to give any actual comparable info.

Boss logs work in raids because it's an even playing field. Everyone starts the encounter at zero - all CDs up, all buffs available, etc. It's the same across the board, for a given difficulty. Same number of players even when it's mythic. That's a good starting point for a meaningful comparison, because you can actually judge people's performance in that kind of setting.

Without that kind of baseline to go off of, what are logs useful for except for individual analysis? And you don't need to publicly log for that if you just want to analyze your own gameplay. Showing it to other people makes preciously little sense in almost any circumstance; and the few where it's warranted can still do it if they like. But a big database of public logs has very little actual value.

We have things like recording comps and times for keys and so on. That's more useful for M+ meta analysis than actual logs would be.

[arkanon]

Several reasons.

Routing, timing, etc. introduce so many variables that it's almost impossible to find meaningful comparisons that logs would be useful for. They're okay for some broad, general questions - but very few people are interested in those.

Key levels also make it so that there's a gazillion categories you'd need to find log comparisons. M+1 to M+27/28/29/however high it'll go. That's almost 30 key levels alone that you'd need to compare across. And then there's affixes. Makes little sense to compare Tyrannical to Fortified, for starters. So double the number of keys again. Want healing logs? Huge difference if it's, say, Grievous week. Even more. And so on and so forth. You'd end up with a matrix so convoluted no one could set it up right to give any actual comparable info.

Boss logs work in raids because it's an even playing field. Everyone starts the encounter at zero - all CDs up, all buffs available, etc. It's the same across the board, for a given difficulty. Same number of players even when it's mythic. That's a good starting point for a meaningful comparison, because you can actually judge people's performance in that kind of setting.

Without that kind of baseline to go off of, what are logs useful for except for individual analysis? And you don't need to publicly log for that if you just want to analyze your own gameplay. Showing it to other people makes preciously little sense in almost any circumstance; and the few where it's warranted can still do it if they like. But a big database of public logs has very little actual value.

We have things like recording comps and times for keys and so on. That's more useful for M+ meta analysis than actual logs would be.

Out of interest, has there been a noticeable change in M+ in the data being discussed? (i cant access the data on this system).

[Biomega]

Out of interest, has there been a noticeable change in M+ in the data being discussed? (i cant access the data on this system).

I'm not sure what you mean by "change in M+". What kind of change? Over what time frame? What context?

[arkanon]

I'm not sure what you mean by "change in M+". What kind of change? Over what time frame? What context?

From Expansion to expansion - has it ALWAYS been something that wasnt logged, or is that more of a new thing?

[Biomega]

From Expansion to expansion - has it ALWAYS been something that wasnt logged, or is that more of a new thing?

It was never something where logging was anywhere near as prolific or abundant as it is in raids. It's probably increased over time simply because logging in general increased; there hasn't really been any big change to M+ where people were like before this logging made sense and now it doesn't. Logging is less useful in M+ purely by virtue of the type of content it is.

[Malkiah]

@Relapses @Biomega @Ryzeth

yeah that was a rhetorical question - the answer(s) are as obvious as they are myriad, and serve to highlight the vast difference in raid logging and the resulting tier lists compared to trying to make tier lists out of ridiculously inconsequential M+ data.