Blizzard on Hotfixing Mythic Halondrus Visual Bugs

[Lumy]

Blizzard on Hotfixing Mythic Halondrus Visual Bugs

After weeks of this being a top priority for our encounter and QA teams, we’ve solved this and fixed it.

Some background

Starting on June 10, in a portion of the Halondrus encounter, multiple abilities began disappearing or rendering inconsistently. While we rolled out some partial fixes a few days later, a number of issues have lingered for the past two weeks, understandably frustrating players attempting the Halondrus encounter.

It’s unusual for a bug like this to take so long to overcome. The first step we take when any bug is reported is to have our QA team investigate and reproduce the bug. Figuring out steps to reproduce a bug is essential, both for understanding what’s causing it and to make it so that we can correctly test the eventual fix. With the Halondrus issues, we ran into an immediate challenge: none of our QA testers or engineers were able to reproduce the bug following the same steps that were causing visuals to misbehave on live realms. That’s extremely unusual, and we spent some time trying to gather more information and test different theories, while applying some quick partial fixes.

Over the last week, guided by the curious case of a single engineer who was able to reliably reproduce the issue (when no one else could), we tracked down the underlying cause as a very niche and previously undiscovered bug in our hotfix system itself. This bug made an unrelated hotfix unintentionally cause player clients to read an erroneous visual “occluder” (an invisible volume usually used to tell the engine that it doesn’t need to render whatever is on the other side of it, to optimize performance) in the Halondrus playspace. The reason internal tests didn’t easily reproduce the issue was-- the actual game data was correct.

The Current Situation

We have now developed a process to both clean up the bad hotfix data and fix the issue that made this possible in the first place. That fix was deployed earlier today.

We sincerely apologize for how long this took to fix. It was a top priority for the team the entire time, but turned out to be almost unique in coming from an underlying cause that took far longer to track down than we ever expected.

[Kumorii]

I like this, mainly cause I'm curious about bugs and glitches in general. Quite interesting to understand why things happens..
Though I imagine it's risky to explain it in detail since it lends to "incompetent" comments. Hope they continue though.

[Aydinx2]

I like this, mainly cause I'm curious about bugs and glitches in general. Quite interesting to understand why things happens..
Though I imagine it's risky to explain it in detail since it lends to "incompetent" comments. Hope they continue though.

There are too many armchair developers in general. I've studied game design and development but I'm not going to comment on how easy or hard something should be to fix for other games, even if they are made in the same engine I use. And WoW in particular is a completely unique thing to develop for.

I'm glad the devs are being transparent about this issue, although to be honest I've had no trouble dealing with the bug.

[cateran100]

There are too many armchair developers in general. I've studied game design and development but I'm not going to comment on how easy or hard something should be to fix for other games, even if they are made in the same engine I use. And WoW in particular is a completely unique thing to develop for.

I'm glad the devs are being transparent about this issue, although to be honest I've had no trouble dealing with the bug.

There is no such a thing "easy" or "hard" in general. Always depends on the abilities and skills of the person who wants to do them (to run 100 meters straight easy, hard or impossible? Or depends on the physical and mental state of the person? Same to solving math quotations or climb to the Mount Everest...)
In short what is easy to somebody turns to be impossible to another...despite being the same thing

[erifwodahs]

There is no such a thing "easy" or "hard" in general. Always depends on the abilities and skills of the person who wants to do them (to run 100 meters straight easy, hard or impossible? Or depends on the physical and mental state of the person? Same to solving math quotations or climb to the Mount Everest...)
In short what is easy to somebody turns to be impossible to another...despite being the same thing

You went too deep on this one mate. People doing the stuff will have general consensus on what is easy or hard. No need to get philosophical about simple issues.

[Pakheth]

Bugfixing always fascinate me because so many things can affect the outcome, not to mention hardware, other software and user error. My SO had to deal with trying to replicate a bug only two players(out of thousands) of the game he worked on had ever experienced, rendering the game unplayable. But he had to eventually conclude he couldn't replicate it without sitting down with the player's own computers, so the bug went unfixed although he spent months on it.

Because of such things I can really admire it when Devs go to the lengths they do to fix bugs like these.

[Advo]

Maybe the fact this alone took weeks to fix is the reason they haven't been able to fix hunter auto-shot bug yet.

[Manhands]

This is a great explanation. I work in QA, and the slowest issues to fix are the ones that are only reproducible in a production environment. For the root cause to be the hotfix deployment process itself, while logical once you eliminate the other possibilities, is also maddening because you tend to trust those long-standing build and deployment systems intrinsically. After all, they simply copy from one location to another, and they aren't usually a problem.

Software is hard.

[Kel_Sceptic]

When the "old employee" that wrote the code leaves the company, and all the rest of the employees need to figure it out from scratch

[tomten]

When the "old employee" that wrote the code leaves the company, and all the rest of the employees need to figure it out from scratch

Code at this level is almost always super documented and commented... Shouldn't take long for a competent dev, let alone a team of devs to figure it out.

[Velerios]

This is an issue of: if it works, don't change it. And blizzards Hotfixes simply worked since years flawlessly. I don't mean necessarily every hotfix, but the deployment. Fixing this kind of stuff can be hellish; nobody thinks of it since it always worked. But maybe it actually don't, and people who had a similar issue before. All in all it fix anotherbug that can get pretty annoying.

[maddhacker24]

I like this, mainly cause I'm curious about bugs and glitches in general. Quite interesting to understand why things happens..
Though I imagine it's risky to explain it in detail since it lends to "incompetent" comments. Hope they continue though.

I second this. Being in SQA myself, it's really cool to find those hard to reproduce bugs. Job well done all around.

[christarp]

Code at this level is almost always super documented and commented... Shouldn't take long for a competent dev, let alone a team of devs to figure it out.

It's really not.

[exochaft]

It's really not.

Yeah, it really depends on the organization. Some are very strict about always documenting their code, some it's like wading through a haystack looking for a needle unless someone is intimate with said code. Sometimes is a combination of the two.

Blizz using blue posts to explain behind-the-scenes used to be a common thing, or at least more common than it is now. Blizz used to have 'watercooler' blue posts to explain what was going on behind-the-scenes, their goals and their processes for fixes or upcoming changes, etc. It wasn't until around Ion took over that these sorts of things stopped... not saying it was his fault, that's just the timing.

From how the blue post is worded, I found it kind of odd that they didn't come up with the hotfix system being the issue sooner. Going back over the years and years of blue posts, there has been many times where stuff doesn't show up on their internal servers but shows up on the player servers, and it tended to point back towards the update system itself and/or the player servers being fundamentally different from their internal server. Maybe this has something to do with who is working there now, as I imagine older devs and engineers would've checked this sort of thing immediately if the internal server wasn't producing the issue since they've had to deal with similar issues in the past. Even outside of coding, when you're experienced with maintenance and problem-solving a system for a long period of time, you kind of get a feel for where your issues are coming from even if they're sort of a curve ball.

[iinverse]

Code at this level should be almost always super documented and commented...

You know the difference

[h4rr0d]

Code at this level is almost always super documented and commented... Shouldn't take long for a competent dev, let alone a team of devs to figure it out.

You are so cute
Remember how they didn’t even have the source code for Vanilla?

EDIT: This is not a jab at you or at Blizzard. It's a jab at the entire software industry. And just to provide some reasoning: Crunch is spelled left right and center in the video game industry. No one (and especially developers) likes doing documentation to begin with. What do you think goes first once there's not enough time to make the deadline?

[Usernameforforums]

You are so cute
Remember how they didn’t even have the source code for Vanilla?

Maybe im misremembering but wasnt that cuz they auctioned off all their old assets?

[iinverse]

Maybe im misremembering but wasnt that cuz they auctioned off all their old assets?

Do you think they store source code on realm servers? How is it possible? )

[h4rr0d]

Maybe im misremembering but wasnt that cuz they auctioned off all their old assets?

Iirc they auctioned off old server blades, that has nothing to do with source code. I'm 100% sure they just didn't think to back it up back then. But that just goes to show that documenting everything isn't SW companies' strongest forte.

[Mic_128]

See, now this is proper communication.