multi-thread in mmorpg server

Question

For MMORPG, there is a tick function to update every object's state in a map. The function was triggered by a timer in fixed interval. So each map's update can be dispatch to different thread. At other side, server handle player incoming package have its own threads also: I/O threads. Generally, the handler of the corresponding incoming package run in I/O threads. So there is a problem: thread synchronization. I have consider two methods:

1. Synchronize with mutex. I/O thread lock a mutex before execute handler function and map thread lock same mutex before it execute map's update.
2. Execute all handler functions in map's thread, I/O thread only queue the incoming handler and let map thread to pop the queue then call handler function.

These two have a disadvantage: delay. For method 1, if the map's tick function is running, then all clients' request need to waiting the lock release. For method 2, if map's tick function is running, all clients' request need to waiting for next tick to be handle.

Of course, there is another method: add lock to functions that use data which will be accessed both in I/O thread & map thread. But this is hard to maintain and easy to goes incorrect. It needs carefully check all variables whether or not accessed by both two kinds thread.

My problem is: is there better way to do this? Or from practical view, the delay can be accepted because nothing will happen between 2 tick (the world is driven by continuous tick, all object will paused in 2 tick's interval) Generally, the interval is about 50ms to 100ms

Notice that I said map is logic concept means no interactions can happen between two map except transport. I/O thread means thread in 3rd part network lib which used to handle client request.

Answer 1

I have consider two methods: Synchronize with mutex. I/O thread lock a mutex before execute handler function and map thread lock same mutex before it execute map's update.

If you do this, then there is hardly any point having separate threads, because only one runs at a time!

Execute all handler functions in map's thread, I/O thread only queue the incoming handler and let map thread to pop the queue then call handler function.

This seems closer to what you should do. The I/O thread should handle I/O - nothing more. It needs to be able to get exclusive access to the message queue or data buffers that let the network exchange data with the game, and that's all.

You would not have 1 single lock for your entire map. Each client has its own lock (or locks), guarding its input and output. Only when that client is sending data or receiving it would you need to lock it.

Of course, there is another method: add lock to functions that use data which will be accessed both in I/O thread & map thread. But this is hard to maintain and easy to goes incorrect. It needs carefully check all variables whether or not accessed by both two kinds thread.

This is not actually difficult at all. The answer is to make the shared data as small as possible. For example, use 2 queues, one for messages coming in (network -> server) and one for messages going out (server -> network). Lock the queue to add or remove messages, and ensure there is no other data shared across the I/O and logic sections.

Answer 2

Here's an architecture you might want to consider:

You might want to use separate processes for these tasks:
- User authentication
- Realm selection
- Each realm (maybe broken into independent maps)
- Ingame subtasks, such as mail/global chat subsystem, item/npc cache, largely independent of other state, that does not need to further burden the map server.

Using different processes ensures that a faulty module won't bring down the whole system. Partial reboot's and hot swapping's inherently supported. It's easier to scalate this model to several different machines if your needs require it.

Each module's internal architecture's very similar: Your input threads just push messages into a queue. Your core module requests this list each update, and feeds its messages to the message consumers. These message consumers run the required actions through transactions - acquire locks on affected objects (player inventory, pickable item), run actions (pick item), free locks. Outbound messages are queued in a message consumer local queue, so no synchronization's required here. Once all incoming messages have been processed, these thread local queues are harvested, merged and its messages forwarded to their respective targets.

Answer 3

You can run your IO on background threads that just waits for a full wellformed packet to come in, and then puts the data in the packet into a queue for the main processing thread to process. The main thread can run in a small while loop like:

while(!shutdown)
{
    dequeue_player_input();
    process_objects();
}

In general a multi-process instead of a multi-thread approach will better suit an MMO design. On MMO's I've worked on, a single process could handle anything from the entire world to a 20x20 meter square of the world. And as player and NPC concentration increased, the server would spin up other processes to handle smaller and smaller pieces of the world. IE: Maybe we'd start with 100 processes spread over 15 machines for the world, then 5000 players would come to see an event in one spot. That one spot alone would be handled by 16 processes none of them guaranteed to be on any single machine.

Answer 4

Break down I/O threads and map threads into DP algorithms.

• Quad/Oct-Trees are good just because they are simple.
• Structure with moving objects (AABB), without having to rebuild a big part of the structure for each object
• Depending on the amount of reliable packets you want to send and the pace of the RPG TCP should be fine. A lot of networking libraries implement TCP basically with UDP allowing reliable/unreliable packets.
• Have a deep study on Synchronization architecture of Croquet Project will help you for better understanding.

Answer 5

I use 2 from your suggested solutions; because my server is c#, this is can be handled very nicely with a lock-less queue of WorkItems which contain a delegate and an array of parameters.

The thread which processes WorkItems can be asleep until a message arrives from the client, or it can always process messages at a update fixed interval. Works very well for me.