How to achieve game server redundancy?

Question

The game I'm developing has a simple game server which stores inventory and hero data for each user, as well as deciding the outcome of some random events. The server interacts with PostgreSQL and responds to HTTP queries. It runs on EC2 and Microsoft Azure, but these hosts sometimes become unreachable for a few minutes, and one week the EC2 server froze every day. What are the strategies to make sure a server is always available?

Non-solution: Multiple servers with their own databases, and the client will try to connect to a different server after a timeout. Of course, their data will be absent if it's a different server.

Solution 1: As above, except whenever player data changes, that change is stored, and each player's data is replicated to the other servers (bidirectionally). The disadvantage is that this solution is "home grown" and prone to mistakes.

Solution 2: One primary server (let's leave sharding for later), and one or more backup servers, with the database being replicated. When the primary server becomes unreachable, a backup server is made into the primary server. Problems: this needs additional technology for directing queries to the right server, and also to decide when a backup server needs to be promoted to primary. Can this "manager" also hang? It sure can if it's on AWS! That would bring the whole system down.

Solution 3: We could pay for database hosting with some guarantee of reliability, and have multiple servers but just one database. We would assume the database is reliable (but still back it up). The user could connect to any reachable server, and the servers would not need to worry about having stale data, since there will be only one database. (Again, sharding is a separate problem, and we could just multiply the whole setup without fundamentally changing its architecture.)

So how is reliability achieved when individual servers aren't reliable? I like solution 3 for its simplicity, if a 3rd party database can be relied upon.

Answer 1

I have done your Solution 3 in the past, and it worked quite well.

We then had an arbitrary number of server machines inside the server farm, each of which was running one or more pieces of the game server software (which we had split up into about ten different programs, each handling different tasks). We had one "Object Request Broker", which clients would query about where to send particular types of network request. If one server went down for some reason, the ORB server would notice after a few seconds, and would send clients to a different server instead. This made things really flexible, in terms of upgrading servers or adding and removing capacity. The only 'fragile' pieces in the system were the data stores (which were a database and a couple of other bits of short-term storage which were also shared between an arbitrary number of servers)

Note, though, that all of this used a physical server farm, not a cloud-based one (cloud-based computing wasn't yet a thing back then). Downtime was almost entirely at our own discretion, apart from infrequent calamities like network outages, power failures, or crashes. If a cloud outage takes all your servers out at once, it doesn't really help you that you had several of them; it's a little bit of different situation where we were in control of the downtime ourselves, and could take machines down and do maintenance on them one at a time so that server remained up pretty reliably even through maintenance periods.

Answer 2

I agree that option 3 is the way to go - this is a pretty standard setup and requirement of resilient web hosting, and not something that you should really have to write yourself.

For full resiliency you'd be looking at a step-up such as:

1. A load-balancer (ideally more than one, on a Virtual IP and fronted by a high-availability service to health check the load-balancers and route traffic as needed).
2. Multiple servers running your Game Server application (again, ideal hosted in different locations).
3. A Database Server (ideally that offers replication, with replication into a second location).

Obviously, these services can get fairly expensive, but AWS or Azure should offer you some free options to get started - both of these offer solutions to create multiple copies of your application either within the same "region" but in different parts of the data centre with traffic automatically loadbalanced or in multiple "regions" and then either balance traffic between them or have one as a "hot standby".

But as you note you could start by using a reliable database host and calling that from all your servers - typically, a reputable host will have a fairly reasonable service level agreement for Databases, so you might be able to get away with a single database server for your application - or you could look into something like MongoDB Atlas which offer a free 3 node replicated instance to get you started.

What you should also do is try and work out why your EC2 nodes are freezing - is it AWS networking/infrastructure issues (for example if you're only running one instance, AWS will have to move/stop-start your application when performing updates to the underlying hosts - scaling out to multiple instances in different availability zones should reduce this) or is it that your application is crashing or hanging (could be any number of reasons, but deadlocks and data connections are typically good starting points, followed by memory leaks).