Greg Street (Ghostcrawler) on "Internal Testing VS Player Simulation"

[A dot Ham]

True, but what I think your quoted poster was talking about (I could be wrong), is all that theory crafting is for those mystic raiders and the "1%ers" if you will, of the game. If spec X does 5% better than spec Y, 99% of the people don't really care that much.

It's like my shadow priest experience. I said I started in BC and was almost always top damage and dps in raids, people would have a shit fit and say I was lying or lying. Well, it is a fucking fact and we certainly weren't top 1% guild and people like to always run numbers of Method or Paragon or what the fuck ever top guilds. Oh, so if Insidia doesn't have a shadow priest in their world first, shadow priests must be shit? Nah bruh.

Current content, I really liked my Survival hunter. /shrug I'm competitive in the groups I'm in. That's all I really care about. I'm not a 1%er.

I was in a guild during vanilla that was a top contender in progression through the beginning of AQ. I joined a much more casual/fun guild and I played a lock through BC, occasionally I would hit up the old guild on alt runs. I don't recall my dps, but my overall damage was always #1.

In WotLK I started playing a priest (back when throughput mattered) it was basically dps meters... for healing, and guess what... I was #1 again. By the end of WotLK i was playing my enhancement shaman... and guess who was #1 dps? Me. I never thought the game was overly hard. Most classes were pretty easy to pick up and play.

At the time the competition (very friendly) between guildies and our performance was pretty close. The disparity between similarly geared dps was NOT what we see today.

You start getting into MoP and beyond, and all of a sudden I go from being pretty comfortable at #1 to being better than average dps, and most of the time being better geared. Elemental shaman.

Now why does my personal performance change? This isn't real life sports... I'm not aging out of the industry. I am not all of a sudden sucking (maybe I am honestly though if that's where your mind goes you're a douche). No the only thing that consistently changes from xpac to xpac is the classes. Blizzard is making the changes. I would rather see ZERO changes to my class and maintain a consistent performance than to have some new gimmick they now need to drag their feet on and fail at balancing.

All of that is really anecdotal... until you start comparing that experience to data logs from other players and from sims. Oh okay... well it isn't JUST me (thank god) all elemental shaman [or insert poor performer here] are performing like shit (except maybe that top tier 1%). So is the game balanced around that top tier? That seems like a terrible recipe.

- - - Updated - - -

I work in a numerically-heavy scientific field with lots of stats/trend analysis, and I can totally see where GC is coming from.

The majority of the WoW player have a shocking lack of critical thinking skills. You can post a graph of sims and about 5% of them will actually ask the important question which is: what assumptions were made here? Similarly, the first people to cry about their class on the official forums are the same people to claim their class is "bottom of the barrel" without referencing or linking the magical logs which would probably prove the opposite.

Really can't blame Blizzard for not releasing any internal data

I think you can.

Most rational people can discern between facts and feels. Making baseless claims gets ignored by blizzard and gets ignored by the rational crowd as well.

The reality is that most people are relying on relatively small group of theorycrafters anyway.

So you have Blizzard (small group) and theorycrafters (also small group) who are having a very one sided conversation, and the rest of the community is just watching the interaction and waiting to see what the theorcrafters do.

You start to wonder why they have enabled us to track our performance at all. Blizzard (all knowing) makes decisions and changes that create an imbalance. This imbalance is perceived because of the testing and data collection and reporting done by said theorycrafters, to which Blizzard responds, "You are wrong."

If you aren't going to share insight into why we are wrong then it would save a lot of headache and heartache if you simply removed the ability to track, thereby removing the ammunition for the conversation.

Of course then we would still have our feels (which is almost more dangerous) because I would be world questing and see a fire mage pop in clear 5 packs of falcosaurs, while just vomiting senseless spam. Meanwhile I'm over here with this single baby falcosaur and I've gone through my rotation twice. (obviously I'm being a little hyperbolic) My point is that even without exact numbers a person can have a pretty good idea how they are performing. Especially when you are side by side other players and you see them kill something in half the time it takes you.

- - - Updated - - -

You seem to know a lot about it. Please share their internal assumptions, describe the player audience that is the target of their simulation, and give examples of imperfection and what was missing from their model that caused the problem.

Lastly, and I mean this sincerely, if you have a deep understanding of how it's all supposed to hang together over 36 specs, thousands of pieces of gear, dozens of different types of fights and all the rest, do a sim and submit it to them with a resume. If it's well enough done, it's probably good enough to land you a job there. Really.

I think you're touching on something here that is a little off topic, or rather could be its own topic entirely.

Does:
http://www.wowhead.com/item=134193/b...ins&bonus=1795
Really need so many versions?

Has that REALLY improved the quality of life for players?

How has the plethora of gear option effected their(Blizzard's) ability to balance correctly/appropriately?

Blizzard has this dichotomy between making things simple (pruning abilities) whilst simultaneously completely muddying the waters with available gear/rewards and how secondary stats play such a pivotal role to performance.

We then rely on websites and mods to optimize our gear. Which is being created by these theory crafters... who blizzard is saying is WRONG about their sims. You get where this is going? Its a vicious circle.

[MoanaLisa]

How has the plethora of gear option effected their(Blizzard's) ability to balance correctly/appropriately?

It affects everyone who is trying to do a sim. Do you simulate and balance around BIS or something somewhere in the middle? The point of all of this is that there is no one condition in the game that is suitable for running a simulation and then balancing everyone around it. Real life play with millions of players muddies the waters to an extent that anyone can find something that doesn't really look balanced. Type of fight, gear, available buffs, food, enchantments, whatever.

Do you balance best case or do you balance more around what average players can do? Pick either one and I can construct an angry rage-filled argument about how stupid you are. And that's just the point. If you balance around BIS on a patchwerk fight then just about every player not in BIS or trying to get down a different kind of fight is going to show a lot of variation and complain about how Blizzard can't balance for anything. I don't think that Blizzard balances around BIS and Patchwerk fights so whatever their goals are it's probably different than 3rd-party sims.

[Thorbalt]

im willing to bet that most sims assume perfect timing in everything, and blizz does not. that right there could royally fuck up the numbers for several specs

[A dot Ham]

im willing to bet that most sims assume perfect timing in everything, and blizz does not. that right there could royally fuck up the numbers for several specs

I see most sims with a best possible and an actual result. I think this is accounted for.

- - - Updated - - -

It affects everyone who is trying to do a sim. Do you simulate and balance around BIS or something somewhere in the middle? The point of all of this is that there is no one condition in the game that is suitable for running a simulation and then balancing everyone around it. Real life play with millions of players muddies the waters to an extent that anyone can find something that doesn't really look balanced. Type of fight, gear, available buffs, food, enchantments, whatever.

Do you balance best case or do you balance more around what average players can do? Pick either one and I can construct an angry rage-filled argument about how stupid you are. And that's just the point. If you balance around BIS on a patchwerk fight then just about every player not in BIS or trying to get down a different kind of fight is going to show a lot of variation and complain about how Blizzard can't balance for anything. I don't think that Blizzard balances around BIS and Patchwerk fights so whatever their goals are it's probably different than 3rd-party sims.

That's kind of my point. Have they themselves created a monster that is nigh impossible to actually balance?

[MoanaLisa]

That's kind of my point. Have they themselves created a monster that is nigh impossible to actually balance?

It's the nature of games like this. Name one MMO with multiple-classes that has ever been "balanced" as rigorously as you are criticizing Blizzard for. Just one.

[A dot Ham]

It's the nature of games like this. Name one MMO with multiple-classes that has ever been "balanced" as rigorously as you are criticizing Blizzard for. Just one.

It isn't the "nature" of it. You make it sound like they created this beast and have no control of it. You are designers/developers... grab it by the balls and make it do what you want it to.

My limited experience with other mmos would not suffice as evidence.

But maybe I just got lucky in the few that I played and picked the OP fotm. I would have to say that in my limited experience with other mmos I never felt like the particular class I was playing was underperforming. I may have felt like a given class was TOO powerful, but I NEVER felt as discouraged as I have with wow the last few years and being unfortunate enough to pick the unloved forgotten step child of WoW.

By the end of WotLK I felt like WoW was in a pretty good place. Its pretty much been a disappointfest since then (in terms of balance). Just to clarify I am mainly talking about PvE.

This is where the homogenization argument actually makes sense. If we aren't going to have niche roles (and we haven't since BC) then each class/spec has to perform relatively equal for that given role, dps, tank, or healer. If you are going to be comfortable with and allow significant performance disparity, you need to flesh out specific niche roles.

[Tharkkun]

It's the nature of games like this. Name one MMO with multiple-classes that has ever been "balanced" as rigorously as you are criticizing Blizzard for. Just one.

Not a single one. Most just throw of the nerfs with no communication or care for how it effects the end game as well. I remember in Rift I was playing the flamethrower spec. All of sudden it got huge nerfs and no one has ever played the spec since. That was almost 3 years ago.

[MoanaLisa]

It isn't the "nature" of it. You make it sound like they created this beast and have no control of it. You are designers/developers... grab it by the balls and make it do what you want it to.

Noted that you didn't bother to name any games or really respond to my specific question at all. That's fine.

Just a point of accuracy: You are the one who is claiming they made this and have no control over it. I said no such thing and frankly it pisses me off no end when people do this.

[A dot Ham]

Noted that you didn't bother to name any games or really respond to my specific question at all. That's fine.

Just a point of accuracy: You are the one who is claiming they made this and have no control over it. I said no such thing and frankly it pisses me off no end when people do this.

Seriously you trying to bait me or what? How does someone who blatantly misrepresents the positions of others or the inability to hold a civil conversation become a moderator?

You responded to another poster and brought up an albeit interesting tangent. I responded with a simple question:

Does:
http://www.wowhead.com/item=134193/b...ins&bonus=1795
Really need so many versions?

Has that REALLY improved the quality of life for players?

How has the plethora of gear option effected their(Blizzard's) ability to balance correctly/appropriately?

You respond with:

It's the nature of games like this. Name one MMO with multiple-classes that has ever been "balanced" as rigorously as you are criticizing Blizzard for. Just one.

Point of fact: YOU are the one making outlandish claims (that are not supported), and I have responded to each of those with questions (not statements) clarifying your position. Not once have I actually made my own claim here.

The only statement I have made was to refute the "nature of the beast" bullshit argument as if it is the natural order of these things to be imbalanced. I don't accept that, because it isn't true. The game is construct, a design, it is not organic, it does not evolve on its own. If the game or its content is imbalanced its because someone designed it to be so, intentionally or unintentionally. Blizzard isn't working with actual magic behind closed doors, a little bit of fel, a little bit of shadow, sharp pointy things for flavor... and *POOF* demon hunter.

Noted that you didn't bother to name any games or really respond to my specific question at all. That's fine.

I did respond, just not the way you wanted.

My limited experience with other mmos would not suffice as evidence.

But if you MUST have it, SWTOR, Guild Wars 2, City of Heroes, (does Maplestory count?), Warhammer, Rift, Aion, Wildstar.

Presumably you will now start an itemized anecdotal list comparing the balance of WoW, to that of each individual game listed.

You should really take a look at this website:
Thou shalt not commit logical fallacies because you do it quite often.

Posing the question or demand to list OTHER MMOs with a higher standard of balance: Strawman
Claiming that I am making a claim you YOURSELF made: ambiguity
The argument you will present to defend/compare WoW against other mmos: Anecdotal

I was having a perfectly fine conversation with you and then you go and get all nasty.

Perhaps you need to go outside and get a breather, take a break. You are losing touch with reality here a little bit Mr. Snarky McSnarkface. To be quite honest I'd report you for trolling... but you're a moderator. Not much is going to happen there... PROTECT THE BROTHERHOOD!

Have they themselves created a monster that is nigh impossible to actually balance?

Note the question mark. Its a question. A dialogue.

It's the nature of games like this. Name one MMO with multiple-classes that has ever been "balanced" as rigorously as you are criticizing Blizzard for. Just one.

I said no such thing and frankly it pisses me off no end when people do this.

Be as pissed off as you want, I didn't misrepresent what you were saying: you DID say it.

[thebelltolleth]

Sigh not going to go into with you on that subject.. that's a really dead horse at this point. He admits those were his ideas and were his projects to lead. This whole idea that development is a think tank/pool and that not one person is responsible for design direction is hogwash. It is a lie even he himself perpetuated while he was at Blizzard, and it wasn't until he was gone that he took ownership of such ideas. The whole point

....[hand in face]


And yet, you just went into it on that subject.

Which is it gonna be?

[rda]

You seem to know a lot about it. Please share their internal assumptions, describe the player audience that is the target of their simulation, and give examples of imperfection and what was missing from their model that caused the problem.

Lastly, and I mean this sincerely, if you have a deep understanding of how it's all supposed to hang together over 36 specs, thousands of pieces of gear, dozens of different types of fights and all the rest, do a sim and submit it to them with a resume. If it's well enough done, it's probably good enough to land you a job there. Really.

I don't specifically know, I am making an educated guess. I have enough experience, however, that if there was a way to organize a bet, I'd put money on it.

The balancing tools that they have you wouldn't even call tools. Typically, you'd have a couple of scripts that would give the general sense of where specs stand related to each other but wouldn't be specific enough to be useful on any particular fight, plus maybe twenty or so purpose-written tests for specific cases (ie, for new fights) which get outdated quickly and then you reuse them for new cases. Sims like on simcraft go for the general case, but make it far more detailed. The level of detail is frequently unreasonable especially for the level of quality of the sim code and so devs scoff at the sims for trying to be penny-wise pound-foolish - rightly. But, as I said, that's half of the story. The devs *DON'T* have any better tools themselves. They just think the sims aren't particularly good. And yes, they are right, sims aren't particularly good. But what they have is at about the same level of good, they just err in a bit of a different direction. My point is that GC does not tell that half of the story, and it's kind of important, because without it you'd think the great devs have tools to balance better than sims, while in reality they don't - that's not an argument for using the sims, let the devs use their scripts, whatever, I am just saying that the result is the same trash that needs constants tweaks for each raid tier.

Do I think that they could make better tools and balance it better? Yes, I think they can make it good enough not to have to fix half of the specs after each raid tier. But it's pointless to be talking about that when they are revamping specs as much as they do every damn expansion. Until they stop doing that, no amount of tools will help, it's just too many changes coming too often, it's the wrong problem to start at.

[MoanaLisa]

But it's pointless to be talking about that when they are revamping specs as much as they do every damn expansion. Until they stop doing that, no amount of tools will help, it's just too many changes coming too often, it's the wrong problem to start at.

Agree 100% with this part of it.

[rda]

Agree 100% with this part of it.

BTW, here's one nugget of info illustrating what I am saying about the tools that the devs have (that they have nothing terribly spectacular):

https://worldofwarcraft.com/en-us/ne...aft-developers

Q. If you have the internal tools for simulation and know they are working properly, why not release them so players can use them?
A. (GC) A couple of reasons. First, there is a long history of theorycraft in this game. I think if we remove too much of the mystery and experimentation, then the game loses some of its charm. There is less motivation to try out different rotations or gems to increase your DPS if you know with near certainty even before you begin what your max DPS is. Second, I feel like we'd have to monitor players using it correctly. Within seconds a bunch of "My DPS is lower than you said it is" posts (whether unintentionally or maliciously) would spring up, which we would then have to troubleshoot or verify.

I mean, sure, this is not "we don't really have much to release", you won't see anyone say that regarding a hot topic like balance (if you do, players will scream at the top of their lungs). But you can see what the reply is - "nah, why break the mistery /giggle" and "it's too complex, you'd need instructions to use it 'properly'". The second part is more or less a direct admission "we don't have much that could be used by people other than us", and that's what devs usually say when they don't want to say that what they have is perhaps too niche / incomplete.

There were a couple other things like that where you could sense about the same reading between the lines and I certainly saw nothing that would give a different sense.

[kamuimac]

What are your thoughts on internal testing vs simulation and player experience? Too often I see "our internal testing doesn't show that" or "our internal testing shows something different" without actually telling players what that is. It completely invalidates a players concerns and first hand experience. If a player is doing something unintended is it not the job of the designer to either design better and redirect a player to play as intended OR accept what the player has done and make that design cater to the unforeseen way the player is utilizing your design?

“What are your thoughts on internal testing vs simulation and player experience? Too often I see “our internal testing doesn't show that” or “our internal testing shows something different” without actually telling players what that is. It completely invalidates a players concerns and first hand experience.”

I have a lot of thoughts on this topic. I’ll try to be brief.

First, I haven’t been impressed overall with simulation as an accurate predictor of real numbers (and most of my experience here is from WoW). Sims are enormously dependent on the skill of the simulator. A small error can produce terrible output and skew the whole exercise. That may not matter to the simulator herself who is more interested in figuring out things like stat weights or novel rotations. But were often linked a stack rank of DPS, almost inevitably with warlocks on top, as proof that something was rotten in Denmark. So when our tested numbers disagreed with sims, we almost always went with our numbers. When we started receiving real data from live, we would shift to using those numbers instead.

Second, thinking about whether or not we should share data is something that still keeps me up at night. I like to default to transparency, and showing LoL win rates is pretty transparent.

On the other hand, I haven’t had good experiences from experiments with showing data to players. Typically, it never actually settles anything. You get accused at worst of lying about the data, or at best about how you didn’t consider X as part of your analysis. Part of the blessing and curse of statistics as an art/science is that you can use different tests and get different results. This doesn’t mean that statistics as a thing is inherently flawed. It just means that you need consistency from test to test and a good justification for why you chose to handle the data in certain ways.

The factor that usually leans me towards not showing the data, is a question of focus, particularly with regard to players giving us feedback. Personally, I don’t want some large percentage of my interaction with players to be taken up with debates about whether or not our data appropriately handles ping correctly, or how many outliers we exclude, or whether we should run the numbers again with a different item set, or whether we divide results by region or how many games it takes to be experienced with a certain champ or whatever. Those are all valid discussion points, but they just don’t feel like the best use of my time interacting with players. I’d prefer to talk about the game and not debate the data. We are pretty confident in our data. We have smart people with a background in data science and analysis and we’re comfortable with their confidence. The value of the community playing armchair quarterback for our data analysis is not nearly as valuable as feedback from the community about what is working or not for them, what’s frustrating, where they’re having fun, and so on. That part can’t be replicated internally (internal tests notwithstanding) and is a large part for why I try to make such an effort to talk to players in the first place.

I don't like to toot my own horn, but I have been the biggest opponent of GC, particularly his post Blizzard employment deification. Mainly because I'm a huge fan of facts over feels. If you like the guy because of his open communication, Great! However, he has taken ownership of both the design direction to homogenize classes and LFR. So if you praise him for the "golden age of WoW" whilst simultaneously opposing those 2 specific features/design direction you are being oxymoronic.

That being said, occasionally Greg makes it really difficult to hate him.

As much as I hated how homogenizing classes took away from the identity of specific classes, I do like that at one point in time whether I was on my lock, mage, DK, shaman, or paladin I could be relatively competitive on the meters.

GC can f... off - hneslty guy has been fired what ? 3-4 years ago - he has 0 current knowledge on anything - quiting his is waste of space on this site

[rda]

GC can f... off - hneslty guy has been fired what ? 3-4 years ago - he has 0 current knowledge on anything - quiting his is waste of space on this site

He wasn't fired, he quit himself, and what he's saying above makes perfect sense, even though he skips some important bits.

[Sevyvia]

I don't like to toot my own horn, but I have been the biggest opponent of GC, particularly his post Blizzard employment deification. Mainly because I'm a huge fan of facts over feels. If you like the guy because of his open communication, Great! However, he has taken ownership of both the design direction to homogenize classes and LFR. So if you praise him for the "golden age of WoW" whilst simultaneously opposing those 2 specific features/design direction you are being oxymoronic.

Ghostwalker was not responsible for the "golden age" of WoW. He stood on the shoulders of giants in that regard.

[Deleted]

He's gone, so lets move on.

Or do we have to remove this line from wiki now. " I benefit from being able to engage the community directly."
Think i'm nuts enough to fill it up with this , I benefit from being able to engage the community directly, except armchair quarterbacks.
Found the article to back it up

Nah, i can't care about this sh.. he's gone, so why bother.
t's just not showing nice stuf to the ones maybe actualy trying to do their best to make something better or help, then call it armchair quarterbacks in a line that just as well can be read as internet heroes.

[dmfg]

If they communicated accurately then 99% of their replies would be either "your sims are garbage" or "this player is terrible at playing his spec", and the community would go into a nerd rage like we have never seen before.

There are players who believe that they are playing their spec at a top level and the only reason they are performing terribly is because their spec is bad/the Sims are wrong/etc, and frankly it's in blizzards best interests to maintain that illusion.

[A dot Ham]

https://worldofwarcraft.com/en-us/ne...aft-developers

Q. If you have the internal tools for simulation and know they are working properly, why not release them so players can use them?
A. (GC) A couple of reasons. First, there is a long history of theorycraft in this game. I think if we remove too much of the mystery and experimentation, then the game loses some of its charm. There is less motivation to try out different rotations or gems to increase your DPS if you know with near certainty even before you begin what your max DPS is. Second, I feel like we'd have to monitor players using it correctly. Within seconds a bunch of "My DPS is lower than you said it is" posts (whether unintentionally or maliciously) would spring up, which we would then have to troubleshoot or verify.

But we basically already have those posts right? People just aren't utilizing the same tools that Blizzard is. So then the response "your results don't match ours" and ending the conversation is just a convenient cop out. Ya that sounds iconically Blizzard alright.

[Deleted]

I don't like to toot my own horn, but I have been the biggest opponent of GC, particularly his post Blizzard employment deification. Mainly because I'm a huge fan of facts over feels. If you like the guy because of his open communication, Great! However, he has taken ownership of both the design direction to homogenize classes and LFR. So if you praise him for the "golden age of WoW" whilst simultaneously opposing those 2 specific features/design direction you are being oxymoronic.

That being said, occasionally Greg makes it really difficult to hate him.

As much as I hated how homogenizing classes took away from the identity of specific classes, I do like that at one point in time whether I was on my lock, mage, DK, shaman, or paladin I could be relatively competitive on the meters.

GC openly spoke out against LFR not so long ago, and how'd he make it different. Try backing up your facts if you like them so much.