How to handle netcode?

Question

I'm interested in evaluating the different ways that the netcode can "hook into" a game engine. I am designing a multiplayer game now, and so far I have determined that I need to (at the very least) have a separate thread to handle the network sockets, distinct from the rest of the engine which handles the graphics loop and scripting.

I did have one potential way to make a networked game entirely single-threaded, which is to have it check the network after rendering each frame, using non-blocking sockets. However this is clearly not optimal because the time it takes to render a frame is added to the network lag: Messages arriving over the network must wait until the current frame render (and game logic) completes. But, at least this way the gameplay would still remain smooth, more or less.

Having a separate thread for networking allows the game to be completely responsive to the network, for instance it can send back an ACK packet instantly upon receipt of a state update from the server. But I'm a little confused about the best way to communicate between the game code and the network code. The networking thread will push the received packet to a queue, and the game thread will read from the queue at the appropriate time during its loop, so we haven't gotten rid of this up-to-one-frame delay.

Also, it seems like I would want the thread which handles the sending of packets to be separate from the one that is checking for packets coming down the pipe, because it wouldn't be able to send one off while it is in the middle of checking if there are incoming messages. I am thinking about the functionality of select or similar.

I guess my question is, what's the best way to design the game for the best network responsiveness? Clearly the client should send user input as soon as possible to the server, so I could have net-send code come immediately after the event processing loop, both inside of the game loop. Does this make any sense?

Answer 1

Ignore responsiveness. On a LAN, ping is insignificant. On the internet, 60-100ms round-trip is a blessing. Pray to the lag gods that you don't get spikes of > 3K. Your software would have to run at very low number of updates/sec for this to be a problem. If you shoot for 25 updates/sec, then you've got 40ms max time between when you receive a packet and act on it. And that is the single-threaded case...

Design your system for flexibility and correctness. Here is my idea on how to hook a network subsystem into game code: Messaging. The solution to a lot of problems can be "messaging". I think messaging cured cancer in lab rats once. Messaging saves me $200 or more on my car insurance. But seriously, messaging is probably the best way to attach any subsystem to game code while still maintaining two independent subsystems.

Use messaging for any communication between the network subsystem and game engine, and for that matter between any two subsystems. Inter-subsystem messaging can be a simple as a blob of data passed by pointer using std::list.

Simply have an outgoing messages queue and a reference to the game engine in the network subsystem. The game can dump messages it wants sent into the outgoing queue and have them automagically sent, or perhaps when some function like "flushMessages()" is called. If the game engine had one large, shared message queue, then all subsystems that needed to send messages (logic, AI, physics, network, etc.) can all dump messages into it where the main game loop can then read all of the messages and then act accordingly.

I would say that running the sockets on another thread is fine, though not required. The only problem with this design is that it is generally asynchronous (you don't know precisely when the packets are being sent) and that can make it hard to debug and making timing related issues appear/disappear randomly. Still, if done properly, neither of these should be a problem.

From a higher level, I'd say separate networking from the game engine itself. The game engine doesn't care about sockets or buffers, it cares about events. Events are things such as "Player X fired a shot" "An explosion at game T happened". These can be interpreted by the game engine directly. Where they are generated from (a script, a client's action, an AI player, etc) doesn't matter.

If you treat your network subsystem as a means of sending/receiving events, then you get a number of advantages over just calling recv() on a socket.

You could optimize bandwidth, for example, by taking 50 small messages (1-32 bytes in length) and have the network subsystem package them into one large packet and send it. Maybe it could compress them before sending if that was a big deal. On the other end, the code could uncompress/unpackage the large packet into 50 discrete events again for the game engine to read. This can all happen transparently.

Other cool things include single player game mode that reuses your network code by having a pure client + a pure server running on the same machine communicating via messaging in shared memory space. Then, if your single player game operates properly, a remote client (i.e. true multiplayer) would operate as well. Additionally, it forces you to consider ahead of time what data is needed by the client since your single player game would either look right or be totally wrong. Mix and match, run a server AND be a client in a multiplayer game -- it all works just as easily.

Answer 2

I need to (at the very least) have a separate thread to handle the network sockets

No you don't.

I did have one potential way to make a networked game entirely single-threaded, which is to have it check the network after rendering each frame, using non-blocking sockets. However this is clearly not optimal because the time it takes to render a frame is added to the network lag:

Doesn't necessarily matter. When does your logic update? There's little point getting data off the network if you can't do anything with it yet. Similarly there's little point responding if you have nothing to say yet.

for instance it can send back an ACK packet instantly upon receipt of a state update from the server.

If your game is so fast-paced that waiting for the next frame to be rendered is a significant delay, then it's going to be sending enough data that you don't need to send separate ACK packets - just include ACK values inside your normal data payloads, if you need them at all.

For most networked games, it's perfectly possible to have a game loop like this:

while 1:
    read_network_messages()
    read_local_input()
    update_world()
    send_network_updates()
    render_world()

You can decouple updates from rendering, which is highly recommended, but everything else can pretty much stay that simple unless you have a specific need. Exactly what kind of game are you making?

Answer 3

However this is clearly not optimal because the time it takes to render a frame is added to the network lag: Messages arriving over the network must wait until the current frame render (and game logic) completes.

That's not true at all. The message goes over the network while the receiver renders the current frame. The network lag is clamped to a whole number of frames on the client side; yes- but if the client has so few FPS that this is a big deal, then they have bigger problems.

Answer 4

Network communications should be batched. You should strive for a single packet sent each game tick (which is often when a frame is rendered but really should be independent).

Your game entities talk with the network subsystem (NSS). The NSS batches up messages, ACKs, etc and sends a few (hopefully one) optimally sized UDP packets (~1500 bytes usually). The NSS emulates packets, channels, priorities, resend, etc while only sending single UDP packets.

Read through the gaffer on games' tutorial or just use ENet which implements many of Glenn Fiedler's ideas.

Or you can just use TCP if your game doesn't need twitch reactions. Then all the batching, resends, and ACK problems go away. You'd still want an NSS to manage bandwidth and channels, however.

Answer 5

Don't completely "ignore responsiveness". There is little to gain from adding another 40ms latency to already delayed packets. If you're adding a couple of frames (at 60fps) then you're delaying the processing of a position update another couple of frames. It is better to accept packets quickly and process them quickly so you're improving the accuracy of the simulation.

I've had great success optimising bandwidth by thinking about the minimum state information needed to represent what is visible on the screen. Then looking at each bit of data and choosing a model for it. Position information can be expressed as delta values over time. You can either use your own statistical models for this and spend ages debugging them, or you can use a library to help you out. I prefer to use this library's floating point model DataBlock_Predict_Float It makes it really easy to optimise the bandwidth used for the game scene graph.