Build of the Month
Recent Blue Posts
Limited PvP -> PvE Free Character Transfers
The Cataclysm Classic Pre-Expansion Patch Arrives April 30
Recent Forum Posts
Why is it so hard to find a Mythic raiding guild that isn't full of toxic people?
Did Blizzard just hotfix an ilvl requirement onto Awakened LFR?
What's the state of PvP like today?
WoW, Heroes, Battle.Net, and Overwatch engineers talked and answered questions. This was not included on the Virtual Ticket. If you enjoyed it, let Blizzard know!
Originally Posted by Blizzard Entertainment
- The Overwatch team was putting together a demo of Overwatch in China, but they had forgotten they couldn't show anything with blood. The game had to be patched over weak wifi in China to get rid of the blood. The team realized that there was one point in the code where the client determines if you shot another unit or the environment, a simple if statement. They realized they could change one byte of the executable to get rid of the blood. They wrote an executable to load the game client, find the byte, flip it, and save the modified client. It worked and Chinajoy got their Overwatch demo.
- At Warlords launch, things were a little rocky. The team had all of the hardware they needed, but because WoW is an older game, each blade had to be dedicated to certain parts of the game. The team set up the blade configuration to what they thought would work well, but they weren't very particular about what kinds of blades were in each pool. They didn't have enough of the high memory blades allocated to instances, so they were running out of memory. Running in your garrison would spin up a new instance, resulting in hundreds of thousands of instances. It took the team around 8 hours to go into the datacenters and pull blades, add memory, and move blades between clusters. This result in some downtime, but when everything came back up it was much more stable. After that there were just software issues.
- The Heroes of the Storm engine is the Starcraft engine, so when the team wants to make changes they have to consider all of the user created mods that may be using the functionality. They can't change things that would harm user mods in the Arcade. The team built lots of chains and shackles but everyone is okay with it. Thankfully no one leaves Team 1, so all of the people working on it are familiar with things.
- The original WoW developers decided that there would be an array to hold your inventory. The first several entries are things that end up on the paper doll, your head and leg slots and such. After that comes your inventory. At some point they wanted to add a bank to the game, so they added that to the end of the array. Players shouldn't be able to access their bank anywhere in the world, as it would break the code. This was handled by adding lots of statements in different places in the code, defining what the array position was where the inventory ended and where the bank begins. This value was hardcoded all over the place, but it isn't just a simple search to find them all. Some math logic may rely on it being constant. If you want to add slots to the backpack, who knows what you are going to break? It becomes very buggy and error prone, and likely that you are going to make a mistake. This mistake would break the game in a way you don't like. As a result, they would have to put amazing engineers that could be working on new features on a task to look through thousands of lines of code to find all of those cases, as well as the QA department testing every edge case in the world to ensure the change doesn't break anything. This is how we ended up with a fixed size backpack.
- The team would rather fix things when they come back to work on related new features. When they worked on cross-realm zones, they rethought how they handed off players between realms. They ripped out lots of the old code and redid it, resulting in being able to add features in the future without having to worry about 16 slot backpack style bugs. This allowed for seamless instancing for garrisons, along with some new things in Legion that have yet to be revealed.
- All of the games are like legacy code that the Battle.Net team has to support. When the team wanted to increase the number of friends you could have on Battle.Net, they had to go and find all of the places where there was a constraint in the games. The UI in all of the games may have constraints on how many friends you can have and see. All of the games have to be patched to support any new features or changes. The team can never do anything that would require all of the games to patch simultaneously, so changes have to be backwards compatible to some effect.
- It's reasonable to assume the WoW team is looking at new technologies, including DirectX 12. No timeline or announcements yet though.
- Heroes of the Storm is a very data driven game. The design team comes to the systems team and wants to do something fun like a hero that is controlled by two different players (and show it off at Blizzcon). The team panics a little bit, thinks about it, and then gets it done.
- Engineers don't ever want to tell design no. Designers are amazing at what they do and have great creative ideas, so no one wants to tell them no. Especially when the designer will find their own way to do it. There is a picture that gets passed around of a trash can with a iron sitting on top of it. The iron is turned on and there is a coffee pot on top of it, with spaghetti in the coffee pot. Problem solved, they can make spaghetti.
- It is really important to figure out where the designers feel pressure when designing the game and try to make that pressure go away. As soon as you have a designer looking at a dialogue box with 128 checkboxes, they are no longer thinking about the awesome experience they want to deliver, they are thinking about what the difference between LEAVE_WORLD_CANCELS and ENTER_WORLD_CANCELS. (Where is the player when they are not in the world?!)
- Overwatch has strike teams, where lots of people from different disciplines come together to work to get a feature done. With Overwatch, that is usually a hero. They get a gameplay engineer, engine programmer, designer, animator, and effects artist to make Hanzo. They have to bring in the Sound team which is always stressed. Everyone works together to perfect the gameplay, get feedback from each other, feedback from the entire team, and finally feedback from the rest of Blizzard and beta players. A lot of this interaction comes from sitting down and playing through what you are working on together.
- There are two types of heroes in Overwatch: D.Va and not D.Va. They only started working on her a few months ago, not realizing it would take all of those months to get her done.
- The team decided that the designers would eventually ask for something like a rocket that you fire and then control, so they got ahead of it and wrote some code to do that kind of gameplay. Junkrat came along and they thought they had it covered and everything was great. It turns out that all of that had to be redone for D.Va. Bugs came up like jumping out of the mech, but it is still behind capturing the payload, moving along. Another bug resulted in the camera going to 0,0,0 when you jumped out of the mech, leaving you with controlling nothing.
- Engineers and designers have complementary goals. The designer's goal is to ensure that the highest number of players get the best experience. The engineer's goal is to ensure that no one has a bad experience. Think about all of the edge cases and make sure your code can cover them.
- Appear Offline not being ready comes down to some legacy code. It is hard to design it in a way that remains consistent throughout the games. It isn't really hard technically to do. What do you do when your friend is appearing offline but they are in front of you in the game world? Appear Offline really means "I want to hide from my friends so I can play D3 rather than go and raid", but when you are both standing in front of the auctioneer...
- The team uses lots of standard industry tools like VTune, as well as internally developed profilers. There is lots of automation to test things, like putting 10 heroes in a one lane map and having them fight each other to compare performance over each build. Heroes get changed from build to build, but it gives you a general performance trend and catching catastrophic mistakes.
- Overwatch uses similar profiling tools.
- Any time you are optimizing code there are trade-offs. Your knowledge about optimizations, the time it takes to write optimizations, how maintainable your code is going to be, and what constraints you are going to put on the engineers and content team to solve problems.
- The team has had to make tons of optimizations that took the form of talking to artists to simplify collision geometry. There is no flat surface in a body, so optimizations are needed for performance.
- When the team is trying to solve a design problem, they consider the fantasy they are trying to create, not what is the solution they are trying to write. They want to consider the player experience they are trying to craft.
- At one point in his previous life as a HFT engineer, @kurtismcc saved 120ns on a hash function and people were thrilled. It was a big win.
- The WoW team spends a lot of time looking at if it is faster to read static data out of a table to speed things up by 4-5x, then pat themselves on the back. Then they will do something like launch Warlords, spin up hundreds of thousands of instances, and find out that when you want to shut an instance down you have to crawl over a linked list to find the one you want to shut down. Someone decided hash tables would take too long so they put it in a linked list.
- The teams have likely written five our six profilers to meet the needs of each team.
- Blizzard has identified that they really want to make games stay online longer. They want to think less about engineering like something they throw over the wall and more along the lines of achieving three nines. They have been trying to achieve this internally, especially with WoW, but they are dealing with an older live infrastructure. Battle.Net also has a lot of older infrastructure. The team recently tried to get to a one hour patch cycle and they got down to 1h 45min in NA and 53 minutes in EU. They are getting faster and it would be nice if they could live patch WoW without maintenance one day.
- There is a lot of static data that drives the game, so patch days require copying a lot of that data in very inefficient ways to a lot of places. They are working on improving that process. A lot of maintenance is inefficiently pushing updated code, binaries, and data to remote data centers. After that they need to do live testing, as until recently testing environments haven't been as great as they should be. They also have to take down and bring back servers in a specific order. The team has development environments where they test everything, but that all has to be replicated in the live environment and hope testing accounted for everything.
- There are lots of people that enjoy VR at Blizzard, but they work on amazing game ideas, not adding amazing tech without a good idea. When and if they come up with a good application for VR, they will explore it. If VR will help player immersion it would be cool.
- The team uses Perforce on the Overwatch team, as well as proprietary source control software they wrong for art assets.
- Battle.Net uses Git for all of their code and Jenkins.
- The WoW server team uses Subversion still, but there is talk of moving to Git where there is lots of shared code.
- Team 1 uses Git, Perforce, Jenkins. (Not Leeroy)
- The WoW UI team looks at the source code to figure out what functions do and occasionally use WoWPedia. There is some documentation for server side Lua, but client side not so much.
- The Lua engine that runs on the server side is worse than not having documentation, as it actively scrapes the code looking for new functions to add so that designers have more flexibility. This means you end up with 17 functions named teleport and having no idea which one is the right one. If it is bad for addon developers, it is much worse internally.
- WoW doesn't use a ton of agile development. A few of the teams are moving to scrum like things. Blizzard doesn't have a lot of consistency between teams internally.
- The Overwatch team was worried about datamining. We found the Progression system 2.0 and even though you can still see it in the Blizzcon demo, it was something they killed 2 days before beta. They realized they didn't like it at all. They did it on such short notice that they left behind some remnants.
- The WoW team knows that as soon as they put data on the CDN we will pull it and go over it with a fine tooth comb. When they were talking about changing over the system to the new content delivery system, they talked about encrypting every single thing to keep anticipation as high as possible. Encrypting everything would mean that they can't patch them, so you would have to completely redownload that content, so the team decided not to encrypt everything.
- It is challenging to create a locked down and secure environment that allows for rapid iteration. Iteration is very important in both engineering and game design, so it is sometimes given priority over locking things down.
- Blizzard games operate at a scale where an innocuous decision will come back to bite you really hard. You want to let players do the /who query in WoW to see who was online, but players were hitting the server really hard. Servers were spending 85% of their time answering the queries because there were so many players online. A good example of the n squared problem. The team has to be sensitive to things like this, as there aren't population bounds like the other games have.
- Legion is rewriting a lot of code so that you subscribe to changes and get them pushed to you, resulting in more responsive servers. During Brewfest, the ram racing quest wasn't very fun, because you could never see the buckets. The server would only update your area of interest every 5-6 seconds, so you would arrive to the bucket's location before it updated and loaded the bucket. The team is trying really hard to improve that in Legion, but it is a lot of work.
- Team 1 - SC2 and Heroes, Team 2 - WoW, Team 3 - Diablo, Team 4 - Overwatch, Team 5 - Hearthstone.
- The teams all want to be on mobile more. Hearthstone has shown that Blizzard can do mobile well. Lots of people wanted to do Garrision missions on mobile devices and the team is aware and wanted that too.
- Blizzard does code reviews across most of the teams and is taking advantage of modern C++ features to make code more readable. Readability and maintainability is an important value for them. Products have to live for a really long time, which is why it is important.
- There is a cheap solution and a right solution and the team supposed to do the right thing, even if it takes longer. You don't want to have to come back and fix things that were done poorly.
- There is a tension between wanting to understand code vs rewriting code that might look messy. If there is code with clearly defined ways to extend it, that is code that is great, even if it is legacy code.
- Overwatch had a lot of people signing up for a constrained beta test. One of the things they want to see is how the code works in different network environment. One person at Blizzcon was playing from Chile and he let them know that it was playing great. The team has even trying playing over a tethered phone and it worked well.
- Open beta weekends are good for crashing servers and exposing bugs. The D3 beta weekend exposed a bug that sent lots of games to a D3 server after it came back from a crash because it appeared to have no load.
- Technology choices that you can get off of the shelf are generic and don't always solve the problems Blizzard encounters when developing games. The team would rather work with constraints they have built themselves rather than constraints someone else built into a tool.
- If there was open source code to solve a problem on the WoW server team, they would use it. If you look at the legal descriptions in the client you can see a lot of the open source licenses from libraries that they use. They try to not reinvent the wheel.
- Having a completed game is a valuable experience if you are looking for things to put on your resume. Passion for learning and variety in what you have done is also good.
Site Navigation
vBulletin Message