Structuring server-side networking with entity-component systems

Question

I've been working on an online game, and recently have been working on converting the base of the game to use the Artemis Entity System Framework. I'm having a bit of difficulty conceptualizing exactly how the server networking should be changed to work with this approach.

My original approach was to have each session receive packets, attach session information to them and then push them to a master queue. I then had a seperate thread (let's call it InputHandler) which was responsible for removing packets from the master queue and processing them. During the packet handling, this thread can add packets to be sent to clients to an output queue. I then have another thread (let's call it OutputHandler) read from the output queue and dispatch the packets to the appropriate sessions (for example if a packet needs to be sent to all sessions or a single sesion).

My initial idea when converting to use Artemis was to have yet another thread (WorldProcessor) take care of doing all the entity processing (this is where AI, general logic, automatic saving, etc. would be done). However I realized that I could no longer do things such as adding entities from the InputHandler thread as it could cause concurrency issues.

Would a better idea be to do the world processing in the InputHandler thread? I thought this could maybe be done by having a Networking system which would try to handle as many packets as possible during a given time frame and then pass control to the other sytems. Another approach I considered was having the InputHandler add events to a queue in the WorldProcessor which would then process these events during the system processing, but I feel this is adding another layer of complexity and that there must be a better way.

I was hoping you could give me some insight or some tips on how to go about connecting the networking and the world processor in a thread-safe fashion.

Answer 1

There is no need to separate network reading from writing. It's pretty common to have some asynchronous process that reads from sockets in a non-blocking manor and checks whether there is data to send and transmits it when there is.

In fact, my network code in C++ uses a boost::asio::io_service in conjunction with a thread pool of approximately 4-5 threads to handle thousands of simultaneous sessions. This code basically issues a read operation on a socket with an IO completion callback. When the read has data I can process, it calls the callback handler in one of the 4-5 threads, the data is fetched from the network buffer and if a full packet is received, the handler passes the packet off to the world processor. Additionally, this same process waits for output packets to be placed on the output queue and when data is made available, it calls the callback handler, pops a packet from the queue and sends it over the socket. When the send completes, the packet is removed from the outbound queue and if the queue isn't empty, the process repeats until it is.

The other important thing to note here is that using a single master queue comes with it's own drawbacks. If your server is quite busy, such a queue can become the bottleneck if you're allowing multiple asynchronous reads to complete for multiple sessions and you're trying to write the inbound packets to the same queue.

We allow our session class to hold both an input & output packet queue. The io_service takes inbound packets and places them in the input packet queue and monitors the output packet queue and transmits any packets it finds.

At this point, you could allow your world processor to iterate your sessions, process each session's input queue and add any response packets to the session's output queue. The problem this can present is what if two sessions send packets in quick succession but you process them in reverse, allowing player B to kill player A when in reality player A should have killed player B?

To resolve this, the world processor has a pre-processing step where all session packets are combined from their input queues, packets are sorted in time-receipt order and then processed. This minimizes the locking of each session's input packet queue to avoid contention on receipt of new packets, and it allows you to handle packets as if they were pushed onto a single master queue in the first place.

HTH.