Maintenance and Hardware Upgrades Saturday, Garrison Addons, Nov 14 Hotfixes

4:00 p.m. PST
All North American and Oceanic realms have now had their population caps increased. We are continuing to monitor realm stability, and plan to begin testing the increase on European realms soon. In addition, we’re making progress towards correcting an issue that would occasionally cause players to get stuck on flight paths when flying to their Garrison. Our next update will occur in roughly one hour.

3:00 p.m. PST
Our initial tests to increase realm capacity were successful, so we’ve continued to roll out our testing of increased population caps to additional North American and Oceanic realms. We’re also making steady progress to resolve the issues causing instance, world, and Garrison server crashes in Europe. We’ll provide another update approximately one hour from now.

2:10 p.m. PST
After carefully monitoring server performance for the last few hours, we are now testing raising realm capacity above normal population caps. We’re going to temporarily increase the population cap on a few realms in North America, and observe them closely. Our next update will be in approximately an hour.

1:00 p.m. PST
We are continuing to optimize servers and address additional performance issues. Realms appear stable with the increased population caps. We are monitoring server stability as Europe continues through peak play time and populations increase in the Americas. We’ll provide further updates in approximately one hour.

12:00 p.m. PST
As previously posted, both Europe and the Americas/Oceanic realms went into an off-peak maintenance to upgrade realm hardware, and implement a new Draenor-wide instancing tech to improve game performance. Both regions’ maintenances concluded successfully, and so far we’ve seen very positive results, allowing us to dramatically raise the realm population caps for all realms. We’re going to continue optimizing the in-game experience, focus on the new instancing tech, and hotfix gameplay issues as they come up.

We’re currently seeing a very high demand on the game and game services due to the exceptionally large number of people excited to play Warlords of Draenor. We’re focusing on a few specific issues, one of which is a group of realms in Europe that are seeing increased instance, world, and Garrison server crashes. That’s being addressed as an isolated issue and we’re investigating the cause to ensure it doesn’t occur elsewhere.

We’ll be looking into the possibility of Free Character Transfers for our highest population realms, but first we need to get the service to a place where we feel it’s also able to process a potentially large number of transfers while not adversely affecting service stability.

Our next steps are to continue working on the few performance issues we’re tracking in Europe, and continue maintaining and improving the level of service in the Americas and Oceania. We’ll provide another update in an hour.

11:00 a.m. PST
We’re working to mitigate some localized instance and Garrison server crashes in Europe. We’re still seeing what we would expect to see in the Americas/Oceanic region for high queues on very populace realms, and for Europe there are queues across the majority of realms during their peak play time. We’ll provide a more comprehensive update in an hour on where the service currently stands as a whole, and what else we’re working on.

10:00 a.m. PST
We’re continuing to monitor the realms post-maintenance. Realms are being maintained at their raised population caps, and while many realms no longer have queues, we do expect that high population realms will still see a wait to log in. We’ll continue to provide updates throughout the day.

9:00 a.m. PST
We’re continuing to monitor realms and track any individual issues as they occur. We’re still seeing some issues occurring now that Europe is in their peak playtime, but with greatly reduced frequency compared to before maintenance.

All realms are now set back up to their normal population caps. Queues being experienced now are due to demand beyond what the realms are normally capable of handling, and we’ll be looking into how to reduce them further.

8:02 a.m. PST
All Americas and Oceanic realms are live. We are continuing to monitor these realms for reports of any ongoing stability issues.

The EU website and forums issues are being investigated as are reports of "World Server Down" issues and Garrison accessibility.

Thank you for your continued patience and ongoing reports.

7:10 a.m. PST
All Americas and Oceanic realms are now live. We will continue to monitor these realms closely.

Thank you for your continued patience and ongoing reports.

7:00 a.m. PST
We are in the process of marking all Americas and Oceanic realms live and will continue to monitor for reports of any additional stability issues.

All EU realms are live and currently stable. We are continuing to monitor them for any additional reports stability issues.

6:00 a.m. PST
All Americas and Oceanic realms maintenance and internal testing continues as scheduled.

EU realm rolling restarts are in the process of being completed and we will continue to monitor reports for additional recurrences of realm stability issues.

Thank you for your continued patience. Our next update here will be in approximately 1 hour.

5:00 a.m. PST
All Americas and Oceanic realms are in maintenance and continue to progress as scheduled. We are working as quickly as possible to deploy hardware upgrades and code fixes that are intended to improve both access to the game and experience while playing.

Across our EU realms, we're monitoring reports of scattered stability issues and have been performing restarts to pick up hotfixes.

We continue to deploy hotfixes for specific gameplay issues through the night. These are often fixes for quests that are not working correctly or not completing, as well as unexpected behavior of various game elements.

Thank you for your continued patience. Our next update here will be in 1 hour, unless there is an urgent need.

3:00 a.m. PST
All Americas and Oceanic realms are now in maintenance. We are working as quickly as possible to deploy hardware upgrades and code fixes that are intended to improve both access to the game and experience while playing.

Across our EU realms, we're monitoring reports of scattered stability issues. We've seen further "world server down" issues, and are working to correct that.

We continue to deploy hotfixes for specific gameplay issues through the night. These are often fixes for quests that are not working correctly or not completing, as well as unexpected behavior of various game elements.

Thank you for your continued patience. Our next update here will be in 2-3 hours, unless there is an urgent need.

2:00 a.m. PST
Our EU realms have been up with maintenance completed, and players logged in and playing, for over an hour. We've gotten reports of some 'world server down' issues during that time, and we're looking into the situation.

We will begin maintenance on our Americas and Oceanic realms in one hour, at 3:00 a.m. PST.

1:00 a.m. PST
A few minutes ago, we brought our EU realms out of maintenance. The game is live for all European realms.

We will be closely monitoring the performance of our EU servers as we prepare to perform maintenance on our Americas and Oceanic realms starting in about two hours at 3:00 a.m. PST.

12:00 a.m. PST
We’re entering our fourth hour of scheduled maintenance on our European realms.

On our North American realms, queues continue to decrease as we move past ‘prime time’. We’re also making ongoing settings adjustments in order to improve realm stability.

Over the last hour, a number of new hotfixes have been deployed to address in-game issues.

All Americas and Oceania realms will undergo scheduled maintenance starting in three hours at 3:00 a.m. PST, with the game unavailable until approximately 9:00 a.m. PST.

11:00 p.m. PST
For three hours now, our European realms have been undergoing maintenance that includes hardware upgrades intended to improve the user experience. This maintenance period was previously scheduled to end in about three hours at 11:00 a.m. CET.

As we’re now past the ‘prime time’ of a peak number of players on our North American realms, the number of realms experiencing queues and the size of queues have begun to decrease.

We continue to identify issues and deploy hotfixes to the game.

As a reminder, all Americas and Oceania realms will undergo scheduled maintenance starting in four hours at 3:00 a.m. PST, with the game unavailable until approximately 9:00 a.m. PST.

10:00 p.m. PST
Maintenance on our European realms has continued for two hours, with hardware upgrades proceeding as planned. In North America and Oceania, we’ve been making adjustments to realms settings in order to improve performance and stability.

Update 8:06 p.m. PST
We’re beginning work on the Europe hardware upgrades now that realms have been brought down for maintenance. In addition we’re pushing out code hotfixes we’ve been testing to the Americas and Oceanic realms, and we’ll be applying those same fixes to Europe realms during their maintenance. We’ll continue providing regular updates, but please be aware that most of our efforts are now focused on the Europe and Americas/Oceanic hardware upgrades.

Update 7:07 p.m. PST
We’re making preparations for the Europe maintenance beginning in just an hour, and along with hardware upgrades we will be implementing some of the performance improvements we’ve been testing. One of these is a fairly large change to the way the game world is run by the servers—expanding the phasing tech used for the Tanaan intro experience to the entire continent of Draenor. This will allow us to run multiple copies of Draenor simultaneously, spreading out server population, while attempting to keep guilds and parties within the same copy. We’re currently running tests of this tech on a few North American realms, and have found we’re able to raise realm capacity by a significant amount. We’ll continue to provide updates as we progress through the Europe maintenance, and prepare to apply the same changes on the Americas and Oceanic realms.

Update 6:00 p.m. PST

We’re continuing to work on improving realm and login stability, reducing Character Not Found errors, and hotfixing in-game issues. We’re making progress ahead of maintenance on a couple performance improvements, and we’re in the process of testing those improvements now on a few realms. Maintenance is beginning for our Europe realms in just a couple hours, and we’ll continue providing regular updates here throughout the night.

Thank you again to everyone for your patience while we continue to work around the clock to address performance issues.

- - -

Last night we scheduled maintenance for Europe to implement server changes to help improve realm performance, and scheduled the same for the Americas/Oceanic realms today. With the maintenance in EU we experimented with raising population caps, and we’re continuing to make adjustments up and down to ensure a smoother game experience. Today we’re continuing to hotfix in-game issues, address any login and authentication issues occurring from the high number of players attempting to play, and make any changes and adjustments we can while the game is live. While these changes have made a marked improvement for the in-game experience, we still have a lot of work ahead of us.

As most of you are aware, the demand on the game is extremely high right now, leading to increased queue times on many realms. We’re also seeing some login errors as a result. This is compounded by us needing to temporarily lower the population caps on all realms to help assist in realm stability issues due to a number of in-game factors that are putting greater than normal strain on the game servers. For those in-game now we’re seeing fairly stable gameplay, but with some continued issues when phasing into the Garrison. Even with lowered population caps we’re not seeing performance where it needs to be.

We've been working toward much larger and longer term service improvements, which we’ll be implementing with off-peak maintenance tonight. A large portion of this extended maintenance will be upgrades to server hardware to help cope with the increased demand we’re seeing. We’re optimistic that hardware upgrades as well as some additional fixes and changes will resolve the in-game issues primarily centered around accessing Garrisons. We’re also continuing to test and implement new performance improvements whenever and wherever possible.

We appreciate your continued patience, and we’ll be providing updates here as work progresses. If you use Twitter, we recommend following @BlizzardCS and @Warcraft for updates as well.

I get that this has been a rocky launch and the servers are stressed, you need to take them down for a few hours to resolve some issues. That's fine, we all would rather the game be running all weekend at 100%.

I just wish you would communicate with your fans/customers better than the robotic way you do now.
We can only communicate about what we know to be true. We're doing what we can to communicate as quickly as we can on key issues with everyone, but there's a lot being done on this end that can't be communicated or communicated right away. As you can see by the hotfixes (Rygarius has been posting), we've been sharing those out as quickly as possible. Both Zaryhm and Bashiok (among others on the team) spent time getting more information on things out last night as well. We have people from multiple departments working around the clock to monitor issues and work on fixes including the CS team here on the forums and on Twitter. This is not anything new for us though. It's what we strive to do with every patch and expansion.

It may seem robotic to you, but we want to also make sure that when we are communicating that we're as precise as possible in what we say and don't want to come across glib to those who may not understand how seriously we take resolving these issues with everyone.

We're working and trying to communicate that work to you as we can, but our primary focus is resolving issues as quickly as possible. We are right with you in that we both want you to play and enjoy Draenor and we also want to be able to play and enjoy Draenor. These things are not mutually exclusive. We're on the same side here.

You should always add two hours to your estimate. Then it looks better when you finish early.
We've had that occur many times. Rarely do people come to the forums to talk about it... because they're playing at that point.

Unknown on the population numbers yet being bumped up. Believe it or not, sometimes we do end up waiting to the last minute in the hope we don't have to extend maintenance, but try to inform people right away.

Honest question here - are you given the discretion to answer questions by yourself (or anyone else on the team) or do you guys kind of post by committee to make sure the posting is crafted properly, without giving too much information out, or worse, implying information you absolutely aren't saying. Again, an honest, no sarcasm question into the mind of a CM and how you guys communicate.
It depends on the situation. Delicate high level information gets a lot of oversight to make sure it's correct and appropriate to go out. Other things can be more off the cuff and we're trusted to know where that line is.

Garrisons, Followers, and Outposts

• Mine: Mining carts should now be easier to spot and should contain more than 1 piece of ore.
• Rangari Jonaa and Limbflayer should now spawn in for characters that have selected the Sparring Arena for their outpost in Gorgrond.
• Followers
  
  • Artificer Romuul: Resolved an issue where characters were not receiving the quest.

Classes

• Druid (Forums / Skills / Talent Calculator)
  
  • Balance
    
    • Resolved an issue where the balance cycle may incorrectly force the Druid to stand up.

Quests

• Defense of Karabor: Resolved an issue where characters may not receive completion credit for the quest.
• I See Dead People: Resolved an issue where Bryan Finn may become unresponsive.
• Kaelynara Sunchaser: Resolved an issue where characters were unable to return to their bodies in Spirit Form if they die inside the Jorune Mines.
• Leave Nothing Behind!: Kur'ak the Binder now spawns more frequently.
• Mulverick's Plight: Increased the drop rate for Mulverick's Axe.
• Save Wolf Home: Gronnstalker Korhol, Beastmaster Trokar, and Giantslayer Zhakta should now spawn more frequently. Additionally, resolved an issue where characters may be unable to complete profession quests while this quest is active.
• The Defense of Karabor: Resolved an issue where completing this quest may cause characters to be unable to progress if they're also on the quest for The Righteous March.

Creatures

• Echo of Murmur now spawns more frequently.
• Made a pass on a number of rare creatures to allow multiple players to receive credit for personal loot if they helped defeat them.
• Reduced the density of Squirrels in Nagrand and critters and Icespine Hatchlings in Frostfire Ridge.