6AM to 2PM maintenance

[Onikaroshi]

That one's slated to be a 6 hour maintenance, so it shouldn't be a big issue. It just stinks it starts at 9 PM Pacific; I get home around 6-6:30 PM Pacific. Oddly enough, I'm on the US east coast.

Would be so nice if the maintenance window occurred when I was at work, but it is what it is.

To stop with my off topic rambling: It could be worse, OP; see my example of FF14's maintenance times.

Unfortunately SE works off Japanese times and does all their patching at once if I remember correctly. It'd be like blizz patching EU servers tomorrow 6 am to 2 pm pdt which would be 1 pm to 9 pm UTC, which is like right smack in prime time lol.

- - - Updated - - -

It is funny, no other mmo's I played prior to wow or up until today has had anywhere near the maintenance window that wow enjoys. I always just chalked it up to blizzard fandom, they're the apple of video games and most people think they shit gold and can't do any wrong, so when you're in that position you can take as long as you want to maintain and people will deal with it and continue to pay. Some will even praise it.

What, outside of patch days WoW has almost zero downtime, and if they do its like an hour a week, but usually just rolling restarts. These servers have some impressive uptimes in the long run.

What high traffic MMOs have you played with less?

[Zoneseek]

What, outside of patch days WoW has almost zero downtime, and if they do its like an hour a week, but usually just rolling restarts. These servers have some impressive uptimes in the long run.

What high traffic MMOs have you played with less?

I was going to reply with a snarky "I said its been better over the last number of years" but just re-read and realized I didn't actually type that part, despite thinking it, haha. Likely before you played, wow pretty much used its entire 8+ hour window every tuesday. I played UO and DAOC heavily prior to wow and neither had anywhere near that maintenance time, basically just reboots and patch maintenance...what wow enjoys now.

Back in those days sometimes on patch days it would be well into the evening before the game was back online and playable. Good times.

[elmoe420]

Blizzard has 10 data centers, 20,000 systems, 1.3 petabytes of total storage, 13,250 server blades, 75,000 CPU cores, and 112.5 terabytes of blade RAM as of 2009, all dedicated to WoW. They have to update all the data on each and every one of those 20,000 systems, including instance and cluster servers. There isn't just one single unit for every realm where they just run an update program 200+ times. It's an amazing feat to update all of that within the time frame they use as it is.

Source

I'm sorry but quoting the number of servers or the total storage size has little to do with explaining why Blizzard needs these regularly scheduled periods of unavailability. Plenty of other much larger distributed systems hitting many 9s of availability. I generally have good faith in the quality of Blizzard engineering but this one has left me scratching my head for many years. I would love to see a dev Q&A that actually discusses what steps make up a deployment for them. Maybe a good suggestion for the developer fireside chat at Blizzcon this year.

At a high level I just think there isn't any financial forcing function to drive a significant improvement to WoW's availability. Players subscribe, paying a fixed amount to play each month regardless of actual time spent in game. Unless availability drops to a point that it affects subscriptions there just isn't a mechanism pushing them to be better in this area. If you looked at a team at Blizzard like the Gear Store team, I'm sure they have a much stricter availability SLA they need to hit.

[Onikaroshi]

I was going to reply with a snarky "I said its been better over the last number of years" but just re-read and realized I didn't actually type that part, despite thinking it, haha. Likely before you played, wow pretty much used its entire 8+ hour window every tuesday. I played UO and DAOC heavily prior to wow and neither had anywhere near that maintenance time, basically just reboots and patch maintenance...what wow enjoys now.

Back in those days sometimes on patch days it would be well into the evening before the game was back online and playable. Good times.

I've played since vanilla, I remember those long patch days and the constant 8 hour tues maintenance, which is kinda why its funny when people complain about it now. But we're not talking about downtimes back then (and UO and DAOC did not have near the infrastructure blizzard did, even back than, hell in 2009 blizz had 20,000 separate server systems to keep up and running) we're talking downtimes now, and right now, blizz has some of the most decent downtimes in the business considering the size of their server architecture.

- - - Updated - - -

I'm sorry but quoting the number of servers or the total storage size has little to do with explaining why Blizzard needs these regularly scheduled periods of unavailability. Plenty of other much larger distributed systems hitting many 9s of availability. I generally have good faith in the quality of Blizzard engineering but this one has left me scratching my head for many years. I would love to see a dev Q&A that actually discusses what steps make up a deployment for them. Maybe a good suggestion for the developer fireside chat at Blizzcon this year.

At a high level I just think there isn't any financial forcing function to drive a significant improvement to WoW's availability. Players subscribe, paying a fixed amount to play each month regardless of actual time spent in game. Unless availability drops to a point that it affects subscriptions there just isn't a mechanism pushing them to be better in this area. If you looked at a team at Blizzard like the Gear Store team, I'm sure they have a much stricter availability SLA they need to hit.

Lately, and for at least the last few years, most "scheduled downtime" has consisted of restarts, which doesn't take long and does have good reasons to be done. Obviously over the years they have improved database cleanup, backend updates, and such to be able to be done every few weeks and in an hour. I mean, they could run redundant servers and limit downtime, but why bother.

[Kanga]

Even with that time, I can see it being extended slightly. One of those things. I don't think I've ever seen a patch be up earlier than the time it said.

[Onikaroshi]

Even with that time, I can see it being extended slightly. One of those things. I don't think I've ever seen a patch be up earlier than the time it said.

Every patch in legion has been up sooner than it said. 7.1.5 was up at about 11 CST so 9 am PST, 7.1 was up early too, and I think before I even wake up (usually around 11, night work is fun yo)

[Kazgrel]

Unfortunately SE works off Japanese times and does all their patching at once if I remember correctly. It'd be like blizz patching EU servers tomorrow 6 am to 2 pm pdt which would be 1 pm to 9 pm UTC, which is like right smack in prime time lol.

Yup; with the current maintenance window now, Europe doesn't get hit near as hard. Was aware of why they have this schedule, even if I bicker about it.

[Self Inflicted Wounds]

8 hours is nothing close to all day.

[Dug]

I'll be at work+school anyways so doesn't matter to me.

[Jastall]

I remember when maintenances took this long, if not longer, every single week. They're dropping a pretty big patch, of course there's going to be extended maintenance.

And I recall that the maintenance for 7.1 finished ahead of schedule. So we might play before 2 PM, too.

[elmoe420]

Lately, and for at least the last few years, most "scheduled downtime" has consisted of restarts, which doesn't take long and does have good reasons to be done. Obviously over the years they have improved database cleanup, backend updates, and such to be able to be done every few weeks and in an hour. I mean, they could run redundant servers and limit downtime, but why bother.

Correct, this is basically the same point I was trying to make (maybe not so clearly). There is no reason for them to strive to improve more in this area. Players still pay 15$ each month and even if WoW is down for 5% of the year it doesn't seem to impact subscriptions.

A lot of this thread seems to be people trying to argue that the time is "required" or that "big databases are hard". Essentially trying to label the problem as intractable. The problem actually isn't and if WoW had a business model where a 5% yearly availability drop translated into meaningful financial loss then there certainly would no longer be scheduled downtime like this.

[Varitok]

The updates usually go online a couple hours before that. The 6-2 time is just to be safe.

[Onikaroshi]

Correct, this is basically the same point I was trying to make (maybe not so clearly). There is no reason for them to strive to improve more in this area. Players still pay 15$ each month and even if WoW is down for 5% of the year it doesn't seem to impact subscriptions.

A lot of this thread seems to be people trying to argue that the time is "required" or that "big databases are hard". Essentially trying to label the problem as intractable. The problem actually isn't and if WoW had a business model where a 5% yearly availability drop translated into meaningful financial loss then there certainly would no longer be scheduled downtime like this.

I mean, aside from redundancies there's about zero way to maintain 99% uptime (you can never have 100%). In the MMO world that sort of redundancy is almost unheard of.

OW can update without need for downtime because it can shift server load around at will.

While WoW has gotten more fluid with their server design there is still dedicated realm servers, for instance Zul'jin is still it's own server blade, to update Zul'jin server or cleanup Zul'jin's database, Zul'jin needs to be taken down. You could make it redundant and have a second copy of Zul'jin somewhere that fires up whenever Zul'jin is down, but it's a rather full server and it would cost just about as much to run this redundancy as the server itself, to what end? It would only be used maybe 3 hrs a month.

And if there is no reason to strive to improve, how have we gone from 8+ hour weekly maintenance and 12+ hour patches to 1 hr maintenance every few weeks and an 8 hr patch window which will feasibly be done in only 5-6? How far can down can you really go? At some point, with such large server architecture, you reach a point where it CAN'T go any faster.

[elmoe420]

I mean, aside from redundancies there's about zero way to maintain 99% uptime (you can never have 100%). In the MMO world that sort of redundancy is almost unheard of.

OW can update without need for downtime because it can shift server load around at will.

While WoW has gotten more fluid with their server design there is still dedicated realm servers, for instance Zul'jin is still it's own server blade, to update Zul'jin server or cleanup Zul'jin's database, Zul'jin needs to be taken down. You could make it redundant and have a second copy of Zul'jin somewhere that fires up whenever Zul'jin is down, but it's a rather full server and it would cost just about as much to run this redundancy as the server itself, to what end? It would only be used maybe 3 hrs a month.

And if there is no reason to strive to improve, how have we gone from 8+ hour weekly maintenance and 12+ hour patches to 1 hr maintenance every few weeks and an 8 hr patch window which will feasibly be done in only 5-6? How far can down can you really go? At some point, with such large server architecture, you reach a point where it CAN'T go any faster.

I won't get into discussing strategies for improving availability. It is fun to speculate but there are so many and its impossible to have a truly productive discussion unless we were to have a better understanding of the actual architecture of WoW. I will say that this isn't even a question of achieving 100% availability (at least 2 nines is a pretty standard goal in my experience though). It is more a "wow 95% still really?" because scheduled periods of unavailability particularly of this magnitude are not acceptable to most businesses and their customers. I think Blizzard is about the only place on the planet that can keep me paying 15$ each month and then only guarantee me I can use their service 95% of the time.

As you point out though the actual downtime has certainly improved over the years (actual unscheduled unavailability certainly has). So there must be some sort of incentive. My guess is this has probably been driven more by the desire to improve quality of life for whoever draws the short straw in the deployment lottery that week hah! The point I was trying to make was that the issue here isn't constrained so much by the technical challenge as it is by the desire to invest time and money in fixing it.

[darklogrus]

Honestly it really isn't that bad. Legion has had a lot less 8-12 hour maintenance and a lot less Tuesday where the service was down most of that day then any other expansion.

AS for the worst patch day nobody can beat FF11, check files > download files > install files and if their service hiccuped or farted at any point it would start the whole process over again.

[Onikaroshi]

I won't get into discussing strategies for improving availability. It is fun to speculate but there are so many and its impossible to have a truly productive discussion unless we were to have a better understanding of the actual architecture of WoW. I will say that this isn't even a question of achieving 100% availability (at least 2 nines is a pretty standard goal in my experience though). It is more a "wow 95% still really?" because scheduled periods of unavailability particularly of this magnitude are not acceptable to most businesses and their customers. I think Blizzard is about the only place on the planet that can keep me paying 15$ each month and then only guarantee me I can use their service 95% of the time.

As you point out though the actual downtime has certainly improved over the years (actual unscheduled unavailability certainly has). So there must be some sort of incentive. My guess is this has probably been driven more by the desire to improve quality of life for whoever draws the short straw in the deployment lottery that week hah! The point I was trying to make was that the issue here isn't constrained so much by the technical challenge as it is by the desire to invest time and money in fixing it.

I mean, you make it sound like wow is down more than it is. If wow were to do one 1 hour maintenance every week (which it doesn't, but for ease lets say it does) and 4 8 hour patches (which is also over shooting since most patching is done in under 6) a year it equals about 80 hours in down time a YEAR. In a year there is 8760 hours, give or take. Wow is down for less than 1% of the year, so at WORST on a year with many DDOS attacks and unforseen circumstances, the lowest wow would ever drop in current times would be 97% in a year. Even if they did full maintenance this month AND were down all 8 hours tomorrow they would have a monthly uptime of 98.5%. This is well within reasonable times, especially for a game this size and especially because they easily maintain 99% yearly uptime. (Honestly don't know where you got 95% from)

[Lillyth]

You would think after all these years Blizzard would know how to patch their servers quickly. I'm not saying they will take up the entire estimated maintenance time, but holy hell.

I think WoW is one of the only MMOs I have played that patch days can literally take all day..

Is it because they are on older servers? Just curious why it takes such a long time for Blizzard to patch

Spoiled kid right here.

[jaymzkerten]

95%

So you keep arbitrarily throwing around 95% availability as some kind of solid number. Just some quick math:

365 * 24 = 8760 hours in a year
regular once a week 1h rolling restart: 1*4*12 = 48h (we'll call this 40h because of the next calculation taking out some rolling restarts)
assumed worst case scenario 8 hours for patch day maintenance, with their current patching timeline of every 1.5 months or so (it's never been that frequent before, and rarely ever that long): 8 * (12 / 1.5) = 64h
64 + 40 = 104h
8760 - 104 = 8656
8656 / 8760 * 100% = 98.8%

98.8%, not 95%. Again that is based on some worst-case scenarios that rarely every happen and a patch every 1.5 months, which is unheard of for this game. It isn't 2 9s or 5 9s, but if I were to put in the realistic calculations for the number of hours it's been down for the past year it probably gets into that 99% mark.

[Kanegasi]

hitting many 9s of availability

24 hours in a day * 7 days in a week = 168 hours in a week

8 hours of maintenance / 168 hours = 0.047619 or 4.7619% downtime a week

1 or 100% - 0.047619 or 4.7619% = .952381 or 95.2381%

If Blizzard would perform 8 hours of maintenance a week, which they don't, they would still have 95% uptime.

Is 95% overall uptime at the absolute lowest estimate just not good for you? You're not the only entitled player to bitch about Tuesday maintenance over the last 12 years. I can safely estimate total uptime over the last year to be at least 99%, with total uptime over the last decade "hitting many 9s". Go ahead and prove me wrong.

[Emerald Archer]

Yeah, let's compare downtime once every week (WoW) and once every 18 months or so (GW2).

It's an hour downtime each week.. I feel sorry for you if you can't pull yourself away from WoW for that long.