Take the 2-minute tour ×
Game Development Stack Exchange is a question and answer site for professional and independent game developers. It's 100% free, no registration required.

Imagine a open-world of 500+ players with data changing as fast as 20 updates/player/second. Last time I worked in a similar MMORPG, it used SQL, so obvioulsy it couldn't query the DB all the time. Instead, it loaded all players from the DB to memory as C++ objects and used them. That is, it scaled vertically. Would it be possible to make that server horizontally scalable instead? Is there a database designed to support that amount of updates concurrently?

share|improve this question
    
Why do you want to update player in database 20 times/second? –  Balon Mar 23 '13 at 22:28
    
@Balon that's where I'm confused. If I don't update it in database, just in memory, then I'll have different states between different machines. But I guess DB updates have some huge overhead so it's not really going to work for that amount of updates? –  Dokkat Mar 23 '13 at 22:37
2  
If you really, really think that different machines (or even processes) need 20Hz updates on hundreds of objects then completely bypass the database and use a messaging system directly. But what you really, really think you want is not what you really want. What you want is to have a sane scope of who needs to know what and then have a way to neatly transport objects between scopes on top of that. You should answer the question of why you need 20Hz updates between different machines for great answers, someone might think of a new way to look at the problem. –  Patrick Hughes Mar 23 '13 at 23:41
    
@PatrickHughes I don't know what I need, I'm just explaining how the game works. Characters move 2~3 tiles/second. A player hunting might be surrounded by a few monsters, so at least 10 tiles/player/second. Then there are items decaying on the floor, on the player's backpack. There are attacks moving in direction of the player, there are attacks moving in direction of the monster. There is the health decaying, the mana being used, timers applying poison damage to the player. So, things change really fast. This is the game design. How could such design be vertically scaled? –  Dokkat Mar 24 '13 at 4:34
1  
I saw this on HackerNews a little while ago: paralleluniverse.co They are working on a database that does all of the spatial segmentation/distribution stuff for you. I would guess that under the hood, they're doing all of the things in the answers below. –  tugs Mar 25 '13 at 23:51
show 3 more comments

4 Answers

Test case of 500 players all communicating, that's 250K streams of information flying around at 20Hz. The internal bandwidth for that would be, assuming 100 bytes each message, about 500MB/sec. Sounds ambitious. Especially between processes.

If you segregate players to groups of 100, that lowers to 20MB/sec, and so on. Which is why MMOs have zones, and in those zones little bubbles of influence, and so on downwards until bandwidth becomes reasonable.

The original problem can be stated that if you have 10 people all sharing realtime information, but you want 500 all sharing, that's an exponential growth of communication links and how can we get around that. I'm afraid that there's no magic bullet that I've ever heard of that can magically make geometric progression go away.

Don't use a database to communicate, that's what messaging is for. Use the database to enforce transactions and store info you don't want players to lose. Most MMOs that I'm familiar with only update the database with dynamic player info every 1-10 minutes, or at handy points like zone transitions or entering "safe" zones in the design.

You may have to redesign the game's need for every player, no matter how far away, to have realtime updates of every other player's backpack contents.

Also change the update pattern from 20Hz to a speed based on distance, someone 1 mile away does not need to know that you moved 1 foot at exactly 230.6 seconds, then another foot at 231.4 seconds, they can deal with you moving 15 feet every 10 seconds.

share|improve this answer
    
Awesome and informative answer, thanks. But I might add that while the world changes at very fast pace, a player can only see other players immediately next to him. I don't see it as geometric - 500 players send info to the server; the server periodically sends info to those 500 players. It's linear, as I see. But the main point is on the 4th paragraph: if I only use the database for storage, then I'm loading data to memory. If I'm loading data to memory in a machine, I'm creating a desynchronized version of the world. That's what I don't get. –  Dokkat Mar 24 '13 at 7:14
    
For 1 client: 1 msg out + 1 msg in = 2. For 2 clients: 2 msg out, 2 msg in = 4. For 3 clients: 3 msg out, 3 msg in = 9. And so it goes. It's like this: send a status msg, server sends result to me and the other 2 clients (1 in, 3 out) and 3 clients all doing that (1 in 9 out). While it looks linear for just one client of the 3, you get to multiply that by all clients for the total throughput of the system. As for desync, even processes on the same physical box are out of sync until the status message is created and sent, it's just a matter of where the pipe empties out, local RAM or net. –  Patrick Hughes Mar 24 '13 at 15:39
add comment

Use area of interest filtering. If a world is broken into 3 servers, and the area on server 1 is nowhere near the area of server 3, there's no reason for them to share information about entities at all.

Likewise, on a single server, only send relevant information to clients. If player A is on the totally opposite end of the map from player B, there's zero reason to send updates about B to A, or vice versa.

When you have multiple servers in a continuous world, you will have entities near an edge on server 2 that are close to entities on server 1. You can send updates from the "authoritative" server for an entity to the other server (when appropriate), and likewise forward any messages to the authoritative server as appropriate.

Yes, in this case, one server will be slightly out of date for particular entities. Don't try to solve that. Just deal with it. Assume that entities may be a little out of date. Do any logic that needs up-to-date information only on the server that authoritatively owns the entities. When an entity affects another, send a message and assume that it may take multiple game logic ticks before it is processed and your view updates.

This design also makes it much easier to thread a single server. No entity should directly modify another, only send messages, and the local per-server/per-thread proxy caches should be assumed to be slightly out of date.

For example, if entity A attacks entity B, don't check the life of B and then send a death message if it hits 0. Just send a "damaged" message, let the authoritative server for B handle it, and then handle any "entity-died" message sent out by server B later if entity A cares about that.

The same applies to any large, scalable non-games application. A central database is not a magical instant-sharing technology. Two servers must communicate with messages, asynchronously, in batches, in order to maintain high throughput. Hence the popularity of technologies like AMPQ and the like. Databases are for storage and support synchronization out of necessity allowing them to be used for communications, not because they are themselves meant for synchronization or communication.

share|improve this answer
    
Thanks, this ended most of my remaining doubts. Also you gave me the idea of separe the servers by players, not areas - this would look smoother. Each servers take care of x players. I really like this! Is this used? And also, there's just one more thing. As I've asked above, I've just learned about a new NoSQL database, Couchbase. It's supposed to be just like CouchDB, except with very fast write/read speeds: up to 200k updates per second! Maybe this could actually work as such a "real-time shared world model", or not yet? –  Dokkat Mar 25 '13 at 0:28
    
I have no idea if that technique is used in the wild besides the usual coarse "sharding" of servers. Just doing it by players and geographical area means each server might need to be aware of a very large number of entities across a diverse set of areas, increasing server load and greatly increasing inter-server communication. Doing it by area means your server might get over-loaded on crowded areas (though you can dynamically split and join areas in that case), but means that each server has a smaller set of relevant non-player entities and geometry to keep track of. –  Sean Middleditch Mar 25 '13 at 2:07
    
@Dokkat: It might be possible to have some kind of "soft areas" where you have each server mainly handling players in a particular part of the game world, but have them transparently hand over the player to another server if they stray too far from their original server's region. You'd just need to make sure the handover is smooth enough that players don't really notice it. You could even try to use some fancy adaptive techniques to keep clusters of interacting players on the same server, even if they happen to be just on a region boundary. –  Ilmari Karonen Mar 25 '13 at 13:36
add comment

You will be probably interested in this article on Gamasutra, where Eve Online developers discuss how it's possible to succesfully run a game with 400 000 active players... in one SQL database.

share|improve this answer
add comment

Don't think of the database as some kind real-time shared world model storing everything about everything at all times — as you've noticed, that can't possibly work.

Instead, treat the database more like an automatically-updated save file: you update the database only occasionally, such as when players log in or out or move from one zone to another, or whenever something important happens that you don't want to be lost in case of a server crash.

The actual real-time world state should be held by the game servers, in memory, just as in your original example. Now, the trick for horizontal scaling is that not every server needs to know everything at every moment. For example, if player A is playing in zone A on server A, server B running zone B doesn't usually need to know what player A has in their backpack — and, if it does need to know that for some reason (say, because player B in zone B casts some kind of remote spying spell on A) it can just ask the other server for that information.

This does require you to assign clear responsibilities to the servers, so that when server B wants to know about player A's backpack, it will know which server has the authoritative information on that. You will also likely want to include some kind of update subscription mechanism, so that e.g. server B can just tell server A "I have someone spying on player A, keep me updated about everything they do until I tell you otherwise." You will probably also want to include some kind of global broadcast system for important global events that players may need to know about no matter where they are; of course, such events should also be recorded in the database, but having them actively broadcast to all server means that the servers won't have to keep polling the database for updates.

share|improve this answer
    
Awesome answer! This was precisely what I was asking, thank you. So perhaps the key is to divide the server in areas, keeping the logic in the on memory. Might I add, though: I've just learned about a new NoSQL database, Couchbase. It's supposed to be just like CouchDB, except with very fast write/read speeds: up to 200k updates per second! Maybe this could actually work as such a "real-time shared world model", or not yet? –  Dokkat Mar 24 '13 at 13:24
    
@Dokkat no, it won't. Couchbase isn't magic. –  Philipp Mar 25 '13 at 9:31
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.