Imagine a open-world of 500+ players with data changing as fast as 20 updates/player/second. Last time I worked in a similar MMORPG, it used SQL, so obvioulsy it couldn't query the DB all the time. Instead, it loaded all players from the DB to memory as C++ objects and used them. That is, it scaled vertically. Would it be possible to make that server horizontally scalable instead? Is there a database designed to support that amount of updates concurrently?

  • \$\begingroup\$ Why do you want to update player in database 20 times/second? \$\endgroup\$
    – Balon
    Mar 23, 2013 at 22:28
  • \$\begingroup\$ @Balon that's where I'm confused. If I don't update it in database, just in memory, then I'll have different states between different machines. But I guess DB updates have some huge overhead so it's not really going to work for that amount of updates? \$\endgroup\$
    – MaiaVictor
    Mar 23, 2013 at 22:37
  • 2
    \$\begingroup\$ If you really, really think that different machines (or even processes) need 20Hz updates on hundreds of objects then completely bypass the database and use a messaging system directly. But what you really, really think you want is not what you really want. What you want is to have a sane scope of who needs to know what and then have a way to neatly transport objects between scopes on top of that. You should answer the question of why you need 20Hz updates between different machines for great answers, someone might think of a new way to look at the problem. \$\endgroup\$ Mar 23, 2013 at 23:41
  • \$\begingroup\$ @PatrickHughes I don't know what I need, I'm just explaining how the game works. Characters move 2~3 tiles/second. A player hunting might be surrounded by a few monsters, so at least 10 tiles/player/second. Then there are items decaying on the floor, on the player's backpack. There are attacks moving in direction of the player, there are attacks moving in direction of the monster. There is the health decaying, the mana being used, timers applying poison damage to the player. So, things change really fast. This is the game design. How could such design be vertically scaled? \$\endgroup\$
    – MaiaVictor
    Mar 24, 2013 at 4:34
  • 1
    \$\begingroup\$ I saw this on HackerNews a little while ago: paralleluniverse.co They are working on a database that does all of the spatial segmentation/distribution stuff for you. I would guess that under the hood, they're doing all of the things in the answers below. \$\endgroup\$
    – tugs
    Mar 25, 2013 at 23:51

5 Answers 5


Test case of 500 players all communicating, that's 250K streams of information flying around at 20Hz. The internal bandwidth for that would be, assuming 100 bytes each message, about 500MB/sec. Sounds ambitious. Especially between processes.

If you segregate players to groups of 100, that lowers to 20MB/sec, and so on. Which is why MMOs have zones, and in those zones little bubbles of influence, and so on downwards until bandwidth becomes reasonable.

The original problem can be stated that if you have 10 people all sharing realtime information, but you want 500 all sharing, that's an exponential growth of communication links and how can we get around that. I'm afraid that there's no magic bullet that I've ever heard of that can magically make geometric progression go away.

Don't use a database to communicate, that's what messaging is for. Use the database to enforce transactions and store info you don't want players to lose. Most MMOs that I'm familiar with only update the database with dynamic player info every 1-10 minutes, or at handy points like zone transitions or entering "safe" zones in the design.

You may have to redesign the game's need for every player, no matter how far away, to have realtime updates of every other player's backpack contents.

Also change the update pattern from 20Hz to a speed based on distance, someone 1 mile away does not need to know that you moved 1 foot at exactly 230.6 seconds, then another foot at 231.4 seconds, they can deal with you moving 15 feet every 10 seconds.

  • \$\begingroup\$ Awesome and informative answer, thanks. But I might add that while the world changes at very fast pace, a player can only see other players immediately next to him. I don't see it as geometric - 500 players send info to the server; the server periodically sends info to those 500 players. It's linear, as I see. But the main point is on the 4th paragraph: if I only use the database for storage, then I'm loading data to memory. If I'm loading data to memory in a machine, I'm creating a desynchronized version of the world. That's what I don't get. \$\endgroup\$
    – MaiaVictor
    Mar 24, 2013 at 7:14
  • \$\begingroup\$ For 1 client: 1 msg out + 1 msg in = 2. For 2 clients: 2 msg out, 2 msg in = 4. For 3 clients: 3 msg out, 3 msg in = 9. And so it goes. It's like this: send a status msg, server sends result to me and the other 2 clients (1 in, 3 out) and 3 clients all doing that (1 in 9 out). While it looks linear for just one client of the 3, you get to multiply that by all clients for the total throughput of the system. As for desync, even processes on the same physical box are out of sync until the status message is created and sent, it's just a matter of where the pipe empties out, local RAM or net. \$\endgroup\$ Mar 24, 2013 at 15:39

Use area of interest filtering. If a world is broken into 3 servers, and the area on server 1 is nowhere near the area of server 3, there's no reason for them to share information about entities at all.

Likewise, on a single server, only send relevant information to clients. If player A is on the totally opposite end of the map from player B, there's zero reason to send updates about B to A, or vice versa.

When you have multiple servers in a continuous world, you will have entities near an edge on server 2 that are close to entities on server 1. You can send updates from the "authoritative" server for an entity to the other server (when appropriate), and likewise forward any messages to the authoritative server as appropriate.

Yes, in this case, one server will be slightly out of date for particular entities. Don't try to solve that. Just deal with it. Assume that entities may be a little out of date. Do any logic that needs up-to-date information only on the server that authoritatively owns the entities. When an entity affects another, send a message and assume that it may take multiple game logic ticks before it is processed and your view updates.

This design also makes it much easier to thread a single server. No entity should directly modify another, only send messages, and the local per-server/per-thread proxy caches should be assumed to be slightly out of date.

For example, if entity A attacks entity B, don't check the life of B and then send a death message if it hits 0. Just send a "damaged" message, let the authoritative server for B handle it, and then handle any "entity-died" message sent out by server B later if entity A cares about that.

The same applies to any large, scalable non-games application. A central database is not a magical instant-sharing technology. Two servers must communicate with messages, asynchronously, in batches, in order to maintain high throughput. Hence the popularity of technologies like AMPQ and the like. Databases are for storage and support synchronization out of necessity allowing them to be used for communications, not because they are themselves meant for synchronization or communication.

  • \$\begingroup\$ Thanks, this ended most of my remaining doubts. Also you gave me the idea of separe the servers by players, not areas - this would look smoother. Each servers take care of x players. I really like this! Is this used? And also, there's just one more thing. As I've asked above, I've just learned about a new NoSQL database, Couchbase. It's supposed to be just like CouchDB, except with very fast write/read speeds: up to 200k updates per second! Maybe this could actually work as such a "real-time shared world model", or not yet? \$\endgroup\$
    – MaiaVictor
    Mar 25, 2013 at 0:28
  • \$\begingroup\$ I have no idea if that technique is used in the wild besides the usual coarse "sharding" of servers. Just doing it by players and geographical area means each server might need to be aware of a very large number of entities across a diverse set of areas, increasing server load and greatly increasing inter-server communication. Doing it by area means your server might get over-loaded on crowded areas (though you can dynamically split and join areas in that case), but means that each server has a smaller set of relevant non-player entities and geometry to keep track of. \$\endgroup\$ Mar 25, 2013 at 2:07
  • \$\begingroup\$ @Dokkat: It might be possible to have some kind of "soft areas" where you have each server mainly handling players in a particular part of the game world, but have them transparently hand over the player to another server if they stray too far from their original server's region. You'd just need to make sure the handover is smooth enough that players don't really notice it. You could even try to use some fancy adaptive techniques to keep clusters of interacting players on the same server, even if they happen to be just on a region boundary. \$\endgroup\$ Mar 25, 2013 at 13:36

You will be probably interested in this article on Gamasutra, where Eve Online developers discuss how it's possible to succesfully run a game with 400 000 active players... in one SQL database.


Don't think of the database as some kind real-time shared world model storing everything about everything at all times — as you've noticed, that can't possibly work.

Instead, treat the database more like an automatically-updated save file: you update the database only occasionally, such as when players log in or out or move from one zone to another, or whenever something important happens that you don't want to be lost in case of a server crash.

The actual real-time world state should be held by the game servers, in memory, just as in your original example. Now, the trick for horizontal scaling is that not every server needs to know everything at every moment. For example, if player A is playing in zone A on server A, server B running zone B doesn't usually need to know what player A has in their backpack — and, if it does need to know that for some reason (say, because player B in zone B casts some kind of remote spying spell on A) it can just ask the other server for that information.

This does require you to assign clear responsibilities to the servers, so that when server B wants to know about player A's backpack, it will know which server has the authoritative information on that. You will also likely want to include some kind of update subscription mechanism, so that e.g. server B can just tell server A "I have someone spying on player A, keep me updated about everything they do until I tell you otherwise." You will probably also want to include some kind of global broadcast system for important global events that players may need to know about no matter where they are; of course, such events should also be recorded in the database, but having them actively broadcast to all server means that the servers won't have to keep polling the database for updates.

  • \$\begingroup\$ Awesome answer! This was precisely what I was asking, thank you. So perhaps the key is to divide the server in areas, keeping the logic in the on memory. Might I add, though: I've just learned about a new NoSQL database, Couchbase. It's supposed to be just like CouchDB, except with very fast write/read speeds: up to 200k updates per second! Maybe this could actually work as such a "real-time shared world model", or not yet? \$\endgroup\$
    – MaiaVictor
    Mar 24, 2013 at 13:24
  • \$\begingroup\$ @Dokkat no, it won't. Couchbase isn't magic. \$\endgroup\$
    – Philipp
    Mar 25, 2013 at 9:31

Other responses have done a good job of pointing out how to use a database, and not to use a database for communication. One other aspect that you might look into is to categorize your updates based on how the information needs to be communicated to other entities. Rather than scope communication to servers, you could distribute your messaging and use pubsub mechanisms for communicating updates between entities. For instance, you might treat location differently based on who's close to you:

  • Precise, real-time location might be useful within radius R
  • Less-precise, less frequent location updates might be useful within radius 2*R
  • No location information might be needed beyond radius 2*R

You might communicate location information for an entity by periodically scanning for entities within radius 2*R (or some multiple of that based on update rate and maximum velocity of an entity), and subscribing the entity to the other entity's precise or imprecise location feed.

You could have different strategies for different types of information, group common things into the same message queues, or have different queues for messages that need to go to different entities (or just send them to the broadest set of entities and have messages discarded if they're not useful).


You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .