World of Warcraft News and Raiding Strategies

Riot Ahab here again to share more updates on what we're doing to address East Coast ping problems. By now, you may be familiar with the NA Server Roadmap, which outlines our approach to addressing ping disparity across the US and Canada. You can check it out here.

Today, I want to introduce Riot WizardOTL (formerly Riot WizardoftheLake), who has been busy leading the charge on Phase 2 of the Roadmap: building a dedicated network for League of Legends traffic.

Why are we doing this? Currently, ISPs focus primarily on moving large volumes of data in seconds or minutes, which is good for buffered applications like YouTube or Netflix but not so good for real-time games, which need to move very small amounts of data in milliseconds. On top of that, your internet connection might bounce all over the country instead of running directly to where it needs to go, which can impact your network quality and ping whether the game server is across the country or right down the street.

This is why we’re in the process of creating our own direct network for League traffic and working with ISPs across the US and Canada to connect players to this network.

I suppose I'll ask: "If this all gets done in the most ideal way possible, what will this make gameplay like? Will ping improve? Will gameplay improve even if ping doesn't with the "less data faster" approach? Will people on the west coast notice a change?"
Many players should see improvements in network quality as our new North American network rolls out. These improvements could be less ping time, less network packet loss, and more stable connections (less disconnects). Ping, packet loss, and stability may not all move in the same direction for all players depending on location within North America, their ISP (how their ISP connects to Riot's network).

Ping, packet loss, and stability are not always tied together and can be affected by many things. When it comes to network quality we prioritize stability, reduced packet loss, and ping time pretty much in this order. You can't play the game if you are not connected, you will see more lag with more packet loss, and ping can become a problem when it gets significantly high or is unstable (shooting up and down all the time in the middle of the game).

The game client only reports ping time to players but it also tracks stability and packet loss to Riot (and in client log files) on a near real time basis (though it can take us a day to crunch all the data from all the game clients before we see a troubling trend developing so we are not yet at real time detection of some network problems). Sometimes packet loss will show up in your ping time, usually you will see it jump up and down quickly which often indicates packet loss is happening or if severe packet loss will show up as lag in the game.

Faster ping time is often reported by players to "feel" better and is often something that is reported to be more meaningful the better you are at playing a game. How fast ping is before you feel the difference is likely dependent on the individual but it is generally thought the lower the better (diminishing returns apply, going from 160 ping to 90 ping is more impactful than going from 110 ping to 40). Lower ping time can also help mitigate packet loss since the game can recover missing packets faster. Because the Internet is a loose collection of various private and public networks, ping time is something that will constantly change for players. Some players could see slight increases in ping times while others can see significant reductions as our new network comes online. Our goal is benefit the most players and then troubleshoot ISPs that still have players with poor network quality.

Based on their other posts this is exactly what they are doing. It won't be going across a congested line if it's a line that Riot owns dedicated to League traffic.
Correct, we are going to have dedicated Internet links for just Riot running through the biggest metro areas of the LoL North America region. Key benefits are the elimination of congestion and optimize the distance packets travel (instead of random hops around the Internet). Some of these links are already online in some of the metro areas but players won't see most of the benefit until we also hook up dedicated links in the metro areas to the ISPs players use. We expect a few hiccups along the way, its a lot of network configuration changes being planned. (Doing the same thing in Europe pretty much at the same time, though we have to deal with lot of country issues versus just companies so some things move a bit slower in Europe).

When will this be finished by?
The full network is slated to be built by the end of March this year.

Does this mean that it will be up and fully functional, or just that the network itself will be ready?
The hardware will be functional -- the agreements may take a little extra time (since contracts always do), and there will be extra work hammering out individual hiccups and making sure as many players as possible have their connections pathing correctly to this network.

So basically, the hardware itself is on schedule, but we'll still have to wait a bit while the ISP negotiations go on. So it mostly depends on the ISPs?
The March goal should include getting the agreements done (and we are currently on schedule in that regard), but the back-and-forth on contracts never fails to amaze me on how long it can take!

Am I reading this correctly in that you are setting up what is your own Tier 2 or 1 network?
Exactly, we're focused on peering directly -- technically we may not meet the classification of a Tier 1 network, but you've got the idea.

How do you see what you're doing impacting gaming as a whole? By the sound of it, what you're working on sounds mostly LoL exclusive...but the same principles would hold true for all/most online games. Is this the sort of thing you could then sell/rent to other game companies? Or would this pave the way with ISPs for other companies to get similar systems in place? Is none of that really being considered yet because just getting this in place is more than enough work?
Great question - and the honest answer is, "we don't know." Every game's architecture is different, however we are working with other gaming companies to try and improve stability and make playing games more fun. I hope that helps.

Can you explain why NA servers have not be moved to a more centralized location? More importantly, you recently upgraded your servers, why not take that opportunity to move the servers to central US?
Network quality is the foundational thing in any online service (online being the key word). Once we have a handle on being able to firmly control the quality of the Internet between player ISPs and LoL data centers we can tackle the issue of what is the best location for the servers. With the big changes in the Internet business model going on since 2013 (net neutrality, rise in huge DDoS attacks, popularity of online video exploding which has created unprecedented Internet congestion, etc.) we have had to take a step back and really think about the foundation first. Network first is the right long term strategy for players throughout North America.

Is there a list of ISP's you have gotten confirmation of their assistance in this current course of action?
We now peer with Comcast, Charter, Shaw, Telus, and we are in negotiations with Time Warner Cable, Verizon, Cox Communications, CenturyLink, and many other ISPs. We're targeting those that carry most of our players, so odds are your ISP is already on the list of folks we are talking to!

What exactly does "peering" with an ISP accomplish? Specifically, what does the ISP do for you, and vice-versa?
Peering is when you connect directly to each ISP instead of just throwing packets out from one location. This allows us to make sure player traffic takes the best path to the game servers and allows us better control the connection quality for players. So your connection gets faster and you have less packet loss and disconnects.

Are you having trouble finding T1/T2 lines in various parts of the country? Do you use an ISPs internal T1/T2 network to transport the data around the country, or are you financing your own and asking the ISPs to forward the data through that?
No, we have our own all optical network and we are asking the ISPs to send League traffic through that.

(This kind of links to 1.) Will you manage the network yourselves or let an outside company do that? Although on-site maintenance for each line presumably would be handled by the provider, how about overall network management? I know as a company you're constantly hiring, but do you think you'll have the manpower to handle it alone?
We manage it internally. We have a great core team and we are actively recruiting to grow that group.

How did you as a company react when an employee suggested doing this? It's a pretty crazy thing to do and I can't imagine you (as a company) agreed immediately. What made you decide that this was indeed the way to go? (I agree, for what that is worth!)
Honestly, it took us a while to digest and wrap our heads around it. We had to meet with outside groups, and recruit people that have dealt with "service provider" networks versus data center networks. And then we ran some tests and found that this was the right path.

How long do you think it will take for the investment this will require to pay off?
We don't really look at it that way, this is the best for the players, so it is just something we need to do.

I saw in another comment you mentioned how you hope to work with other games companies so that this isn't such a one of thing. A followup question I have to that is why do you think no one has done this already?
I think there was a huge gap for software developers and data center network engineers to get over to understand this problem, and most games don't have the requirements that LoL has. So it really didn't manifest until now. As more games need this type of performance, you may see more companies doing this now.

Reddit Q&A
What does "building a dedicated network" even mean? You can't expect me to believe Riot is going to run dedicated lines across the continent, so what are they going to do? Build a giant VPN? Not to mention that phase three of this plan seems pretty familiar, since Riot said they were relocating the servers before and ended moving them further up the coast. Is the new phase three going to move them to Alaska?
So building a network is exactly what we are doing. We have leased circuits from Level3, Zayo, and CenturyLink, we have bought Routers from Juniper and Alcatel-Lucent, and we built and are operating POPs (Point of Presence) Equinix and Zayo locations across the country. The backbone is up in New York, Chicago, Seattle and San Jose. And we are bringing on LAX, Dallas, Atlanta and DC in the next month or two.

The Backbone is actually up, the hard part is the peering with the ISPs and route balancing. We now peer with Comcast, Charter, Shaw, Telus, and we are in negotiations with Time Warner Cable, Verizon, Cox Communications, CenturyLink, and many other ISPs.

The holidays have slowed us down a bit, but we are working to have the whole infrastructure finished by March 31st of this year.

The take away is that this is real, and we are moving as fast as we can.

Just to let you guys know, we are building a Tier 1 network. Just to give you some context, we have data that shows that some players in the bay area are getting 90ms ping, and they live less than 15ms from the servers! This is because some ISPs are sending the bay area traffic to LA, then to Denver, and then to Seattle and then to the servers. Putting a network in and peering is the only way to fix that. In another instance, Charter ended its contract with Zayo and that made Michigan players take a much longer path to the servers. None of this can be fixed unless we have the network in place.

This is a big project with a ton of people working on it and we are Really excited to see it come on line and see it be a huge win for the players.

Is what you're doing something Blizzard didn't have to do or is different than what Blizzard had to do to stabilize WoW at it's height?
I can't talk to what other game companies are doing, but I can say this is the first venture of this size and scope I have ever heard of.

What I'm afraid of is the ISPs charging a premium to the customer in order to have their traffic routed to Riots Private Network.
Blaaki - we are working with them, and the services that we getting from them do not allow them to do that. We are very aware of this concern and will work as hard as we can to make sure this is not something that can happen.