Originally Posted by MMO-Champion
This question is strictly asking about during your time at blizzard. On a scale from 0-10, 0 being useless and 10 being perfect, how useful are player-made tools like simcraft. For myself, I use them because even though I feel like there could be glaring mistakes/oversights, we have no other way to determine the difference between gear sets. Do you think tools like simcraft are good or bad for the game? I ask you because you have mentioned before that you had the actual data instead of sim.
I’ll answer using WoW as a reference point, but this is relevant to other games as well.
Sims were originally a tool players used to try to solve problems like itemization choices, optimal rotations, or even talent selections. For answering those problems, the simulations can be really powerful tools. They are very sensitive to having good input though. If the theorycrafter misunderstands how a particular mechanic works, or just introduces a bug somewhere, the sim may come up with bad results. Because they are generally a bear to work with, it’s often hard to get peer-reviewed results, so sometimes a player would do a ton of hard work, and it would be taken as gospel. Other times there was a strong enough community of theorycrafters to challenge assumptions and offer improvements. It just depended on the spec/class at that point in time.
Sims struggle a lot more when they’re trying to represent reality. They are good for answering relative questions (”Should I choose a dagger with haste or crit?”). They are weak at answering absolute questions (”What will my damage be on boss #2?”). The reasons for this are probably obvious, but sims generally assume low lag and player perfection. Furthermore, they struggle to capture things like high movement, lots of adds, target switching and so on. They also tend to provide an answer (usually “What is my DPS?”) when DPS isn’t actually the real question on some fights, say one where it’s easy to pad meters, or killing / controlling the adds is the trick to the fight instead of the boss, or when DPS during Phase 1-3 isn’t important, but it’s critical in the brief Phase 4.
Where sims are terrible and cause developers a bunch of pain are when they are used to compare the (already inaccurate) absolute numbers from spec to spec or class to class. When a player would email me a stacked rank table of simulated DPS and ask me when they were getting buffs, I admit that my eyes would sometimes roll. The accuracy of the sim (which I’ve already argued is not accurate) is really dependent on the quality of simulator. If one player is simming Fury Warriors and another simulator is simming Shadow Priests, there is no way to tell if they are handling data the same way or even making the same assumptions.
Developers have enough on their plates just building the actual game, so when presented with data that seemed odd to us, it was easier for us just to say “Something seems off” than to try to debug a third party tool or process.
Logs (i.e. what actually happened in a fight) are generally much more useful than sims in answering questions like “How important is my class for this particular fight?” There are a lot of things you have to consider with logs too (”Did the group overgear the encounter? Is this encounter even a blocker for progression?”) but they are better than sims, unless you’re just asking a question about stat weights. (Source