Handling incoming packets immediately or queuing them

Question

I'm using Golang to write a game server. I was wondering what the advantages of queuing the incoming packets for processing over processing them immediately.

For example (processing immediately) each new connection/client a new goroutine/coroutine is created and within that goroutine a loop that reads from the network socket and has a switch statement that routes and handles each packet.

Or (queuing) each new connection/client a new goroutine is created and within that goroutine a loop that reads from the network socket and passes it to a queue. A second dedicated goroutine for this connection/client has a loop that process the packets in the queue.

Is this a pre-optimization? How are MMORPG's (server side) packet recv/handling handled in principle?

I'm using TCP.

Answer 1

Since you clarified in the comments that you really want to know about TCP, the answer is simple: At this level of performance, it usually does not really matter.

If you aim for a higher level of performance, you'll want to use a different approach with far fewer threads than the ones you suggested.

• Use 1 thread per NIC, handling every single client connected to that NIC. Because of the large number of clients you only do parsing, followed by the minimal preprocessing of the messages necessary to enqueue messages on the appropriate target thread. If you use UDP, this thread will likely also implement heartbeat and some retransmission request functionality as necessary.

• Use multiple threads based on the game's design. Easy targets to split off into independent threads are chat and dungeon instances (if they aren't already on different servers), most everything else depends on the specifics of the game, although inventory is often a candidate for getting its own thread.

The primary benefit of a pre-processing thread for all incoming messages (per NIC) is scaling. Additionally, by moving the parsing and sorting to a different thread this is something you no longer need to worry about.

The downside of a pre-processing thread for all incoming messages is complexity if compared to the single threaded mega-application.

Using multiple threads for the initial processing of incoming messages is undesirable. First, it offers no benefit: A single thread is fast enough to parse everything the NIC and the OS can deliver, unless the parsing function is poorly designed (solution: optimize the parsing function). Second, it does introduce unnecessary race conditions: If 2 players want to pick up the same item, player A's packet may arrive at the server first, but player B's request may end up being processed first. Third, using multiple threads to parse and sort incoming messages ends up requiring more performance, primarily due to light synchronization overhead.

Using a thread per player is undesirable, because it leads to large synchronization overhead.

Answer 2

You will probably want to handle packets as they become available. As mentioned in the comments, OS and network drivers will queue packets, but as soon as your code knows about them you'll want to consume them.

However, processing the effect of a specific message may depend on the purpose of it. If it's a simple ping or other synchronizing message, you may want to respond right away.

But if it's a player action, that needs to be reconciled against other player actions, you may want to queue it temporarily and process all such messages at one time.

This is because of general latency. If Player A performs some task and Player B performs a counter to that, it's possible that the messages will come in in a different order than they actually happened.

So you may consider holding all messages during a given frame (some small period of time to allow for normal lag, maybe 100ms) and process them. Or holding the message to be resolved once you have all messages that came in 50ms before and 50ms after are available.

In the end my recommendation is to receive each packet as it comes in, and respond to it if needed, or put the event it represents into a queue to be processed.