What's the best server architecture for real-time games?

Question

I'm developing a Real-Time game which should hold thousands of players in real-time (FPS like. max 1s lag). What would be the best infrastructure for this?

My idea was using 2 server clusters - one for the Server End (all the computing side) and one for the Database end, where a load balancer is "responsible" for each of the clusters. One main server will receive the requests from the users and send back the IP address of the relevant server that the user can work this.

The database cluster will use database replication for consistency between the databases.

There should be a geographical load balancer as well - so it will assign the regional load balancer to each user for best response.

I'm using .NET + MSSQL for the game.

Thanks!

Answer 1

There is no best architecture without knowing a lot more about your requirements, eg. the sort of interactions between characters, how much data is going to be persistent, and so on.

If you can cope with 1 second of latency, you can probably host 1000 players on a single server without problems - but that actually conflicts with the idea of an FPS as they typically require much lower latency, eg. less than 100ms. But a system that can deal with high latency can afford to do everything via message passing which makes consistency pretty trivial. Providing your logic is quite simple, that is - complex logic gets even worse when you turn it into a message-based system rather than a locked-object system - but it all depends on your application's needs.

Similarly, if you're not persisting much data, you don't need the database machine at all, but without knowing that, it's hard to say. If you're persisting small amounts of data, and perhaps only doing it at the end of a tournament or something, again you don't need a separate database, certainly not a cluster of them. On the other hand if you're not persisting much but reading a lot, that's where the replicated databases can help you - but it also indicates that a relational database may not be the best match for your problem in the first place. Often an in-memory cache is a better solution. Similarly, if there are no transaction-style interactions between characters, then consistency becomes less important. (And if there are only a few such transactions, you can make them a special case.)

In fact, be wary of adopting an RDBMS just because it is the done thing in large systems. Although I personally do approve of using them in online games, it's best to look at your requirements and figure out your persistence strategy from that, rather than taking your preferred database and then trying to tweak it with caches and replication to get it to fit your app. You may find that all you need is offline reporting capability, in which case it's probably best to have a background process logging from your game persistence mechanism into a remote RDBMS somewhere else.

Answer 2

Disclaimer: my game programming experience is based around client-side single player games, but I have a background in web applications (specifically on the Microsoft stack), so that is where I'm coming from with this answer, I feel that much would apply, but without actual testing a real game server its difficult to say how it will apply, but here goes. Know this: I haven't deployed a game server, only webapps.

I would suggest a two (server) tier approach. A database tier and an "application" tier; with the third (presentation) tier being your game client.

Relational databases, are great at querying data, and decent at writing data. The key is to serialize your database writes into manageable size chunks that your cluster can handle. The more advanced editions (Data center/Enterprise) of SQL Server support clustering and replication. I would start by building a small cluster and running some queries against it to see how it works.

In the application tier, if you're doing "zoning" or something similar, you can probably get away without setting up any clusters, and simply setup a server per zone. If your zones become to big, you could setup a cluster for each zone.

You will want to build a serialization process for sending data from application tier --> database tier. The key is to have multiple levels of serialization going on. Something like this:

• Level 1: Save to DB every X seconds, includes critical data:
  • Player Health
  • Player Items/Pickups
• Level 2: Save to DB every 2X seconds, includes medium data:
  • Player locations
  • NPC locations
• Level 3: Everything else, as infrequently as possible

This will keep your writes consistent and predictable, depending on the nature of your game, you could have infrequent database writes. The key is to realize that if your application server crashed, you'd have to come back online from the state in your database, so serializing player inventory every 90 minutes might make players upset.

For reading data, you'll want to load as much as possible into memory in the application tier as possible, then insure that all of your code uses this memory pool, in the background you can synchronize the memory pool with the database. As Joe points out, there will be times when you need "real-time" transactions. By serializing most of your writes, you should still have sufficient IO on your database to do real-time transactions when necessary, presuming sufficient hardware on the database server/cluster.