Tech Infrastructure for Competitive PVP Games
Tech Infrastructure for Competitive PVP Games
Featuring Riot's Head of Infra and Live Ops Zach Blitz and Network Next CEO and tech badass Glenn Fiedler
Welcome to the GameMakers newsletter. Join today to stay informed about current and relevant strategic insights that create success.
🎧 Listen to GameMakers content on Spotify or Apple Podcasts
📺 Watch on Youtube
To developers, corporate strategists/execs, and PMs 👋 ,
What kind of technical and network infrastructure is required to launch a competitive PVP game today?
Well, if you could pick two of the best people in the entire world to ask this question, these two people below might just be those two:
Zach Blitz, Head of Infrastructure and Live Ops at Riot Games
Glenn Fiedler, Founder & CEO of Network Next
After a deep and far-ranging conversation with these two guys, I’ve summarized my interpretation of some of the key takeaways in the following summary below.
It’s All About Latency
The name of the game is latency. This is the key variable that will define the infrastructure including scale-up and operating costs for your game.
Also, latency is still largely applicable only to the games industry. So talking to your cloud provider rep who serves e-commerce likely won’t be a good conversation. They likely won’t understand latency requirements for games.
If you want to build a competitive, multiplayer PVP game giving a player experience with “competitive integrity” then balancing who, where, and how much latency players experience will determine the difficulty and cost of technical infrastructure.
Even further, much of the issue isn’t so much about a technological problem of “can we do this?” Today, latency for high performance can be done, however, it’s more of a problem that is as Zach suggests:
Most of the problem is business or product or game design problems.
For competitive games, in particular, some games like Valorant have premised their key differentiation on the kind of competitive player experience they can offer to their players. Zach speaks of delivering a game with “competitive integrity” meaning “everyone has a level playing field.” Therefore, Valorant seeks to provide very high levels of performance to players - meaning low and balanced latency - and also attempts to specifically eliminate key problems associated with competitive multiplayer games such as “peeker’s advantage.”
Zach on Peeker’s Advantage:
Games like Counterstrike and its predecessors had a lot of it.
When a player peeks or looks around a corner first, because of the way the Internet works, they will see their opponent before the person who is standing there sees them.
So what Valorant’s thesis really was is if we eliminate Peeker’s Advantage to the best of our ability globally, that player would really, really want to play a game where that is true. And for the most part, we’ve done it. But it’s very difficult.
Latency Targets
Glenn suggested that as a general heuristical target, competitive PVP games should try to get to:
Good target for competitive multiplayer PVP Games is < 50 m.s.
Above 50 m.s. studies show that there is reduced engagement and higher churn
<100 m.s. is essential
There’s a different price tag for both of those performance requirements as well as optimizing geographically based on where your players are.
Glenn suggests the right approach is to use data to determine when and where your players are having bad player experiences and then to optimize around that.
The Components of Latency
Based on the conversation with Zach and Glenn, my takeaway was that latency should be thought of in 5 different component parts as I depict below:
Server Infrastructure
Today, it’s a no-brainer to just go with a cloud-based solution from Google/Amazon/Microsoft. Glenn prefers Google, Zach declined to suggest a vendor but did say Riot uses both Google and Amazon.
Glenn suggests he only knows of a few companies that still have on-premise server infrastructure: Valve, Riot, and maybe EA. I know of a few other big companies (console not any mobile unless Zynga is still doing it) still operating fairly significant on-premise facilities.
While Zach indicated that cloud vs. on-premise does not actually have cost savings, both Zach and Glenn suggest cloud is the way to go
I posed a direct question to Zach whether game companies retain on-premise due more to political reasons (not wanting to lose jobs, learn new skills, etc.). However, he suggested instead it has more to do with maintaining stability. He related a story that there are some Disney rides, in fact, still operating on floppy disks for this very reason.
“Middle Mile”
Due to the way Internet traffic is routed in the “middle mile” of the Internet, there’s not much game devs can do besides getting geographically as close to their users as possible
In the public Internet, the way BGP (Border Gateway Protocol) routes traffic can be indiscriminate and is not optimized for low latency
Some game companies, or maybe it’s just Riot, have built their own private Internet to address this middle mile routing problem. Riot has spent about $10M a year investing in their own private Internet called Riot Direct.
Glenn’s company Network Next attempts to resolve the problem of the “middle mile” by creating a Riot Direct infrastructure for game companies who can’t afford their own private Internet to use as a service. Pretty smart huh?
Last Mile
This is the part of the Internet between your ISP’s “head-end” and your house or device
There’s really not much you can do about this part but one new service that will definitely help is SpaceX’s new Starlink service that will connect the entire globe including remote parts of the world with mega-fast Internet. Did you know Elon started his career as a game developer? 🕹️
Due to the speed of light moving at full speed in space vs. 2/3rds speed in fiber, Glenn mentions we may be surprised that some points on the globe may actually get lower latency through Starlink than via fiber! 😲
Geographical Distribution
How should developers think about when to open new servers in new geographies?
Glenn: How many players and what’s the improvement that can be provided?
Zach: Find bad experiences and then fix them.
In regions like the Middle East, there’s an additional challenge in that some countries within the Middle East may not get along and there are geopolitical issues. Therefore distance may not be the only issue to resolve in some regions. The latency between Japan and Korea is another example.
Matchmaking
Don’t forget about matchmaking!
Matchmaking can help by ensuring “competitive integrity” meaning ensuring that players are matched not just based on player skill but also based on relative latency if matched to each other
How Should Small Game Companies Approach Latency?
As a small developer who may not have the resources of Riot or want to invest in their own private network, how should small companies approach latency?
Glenn suggests that small devs should first make sure they have a fun game, without that “nothing else matters.”
Then:
Start off with something like Google Cloud, and then once the game gets bigger, then start to focus on latency problems
Begin to intelligently select the location of servers. Glenn names the specific cities to consider in the video below
Shift attention to matchmaking as per above
Find the points where people are churning out using data and optimize those areas
Look into private Internet solutions like Network Next!
Full Panel Discussion with Glenn and Zach
You should definitely check out the full panel discussion available right here below:
How To Get In Touch
Zach Blitz via Twitter: @RiotZBlitz
Glenn Fiedler via email: glenn at networknext dot com
LILA Games is building one of the most ambitious shooter games ever designed! We are currently hiring aggressively in Bangalore, India. See below.
Visit our website!
LILA Openings (Bangalore or Remote):
LILA Openings (Bangalore Only):