UDP packet reliability and re-sending

Question

Like most people on this exchange concerned with UDP client server connections, I've gone through the Gaffer on Games article about implementing reliability. I understand how the ordering and acks and what not works (I think).

My issue is, some packets need to be re-sent until they are received, such as a packet stating that an entity has been created in the world. Others, such as updating the location of an entity don't need to be re-sent if a newer packet containing location data has already been sent.

What if, say, an entity is created and destroyed within a second. Say the create packet is lost but the destroy packet is received. The client has been told to destroy a non existent entity. If it just ignores this, when the create packet is re-sent, the entity will never be destroyed causing inconsistency.

All of this is per-client, so the actual game code should know nothing about it, only the networking code.

So how should this be handled? Do I flag certain packet types as resend. Do I send send a field in all dependent packets saying they depend on packet number x? Neither seem very elegant.

Answer 1

If the client receives a destroy packet without receiving the create packet, then it should know that it is out of sequence. The client won't have a record of the entity. Queue it until the create packet is received.

As you said, some packet types are sequence dependent and others are not. The client should know the protocol.

Another option is to send the create and destroy as a single event in the same packet if the original create packet hasn't been ack'd yet. This would depend on the protocol you define. You might have an extra bit in your create packet that indicates if the entity has also been destroyed, if there is no other information associated with the destroy event. You might also allow sending multiple events per packet for certain event types like create/destroy or even for all event types though if the data is variable length, then you will need a way to delimit the events in your packets.

If the server sends a new (as in different sequence ID) create/destroy packet but meanwhile the client does receive and ack the original create packet, it should know from the entity ID that it's a duplicate, regardless of the sequence ID being different. So it would just ignore the new create event and apply the destroy only.

Answer 2

You should consider using a mix of TCP and UDP flows and add a layer of abstraction over a raw UDP packet like a "message." Then, similar to the Steam API if you're familiar with it, you can expose functions like sendReliable(message, destinationClient) and sendUnreliable(message, destinationClient). Reliable messages can be passed through the TCP stream, and yes you should in fact use TCP streams if the data being sent needs to be reliable or maintain order. Turns out implementing a reliable message protocol in UDP will basically be implementing a worse version of TCP.

You then can decide which data is sent in which way. You're right to send stuff like position updates over UDP, only the latest update is useful and if one is lost there's no reason to resend it. Anything that requires order and/or reliability definitely should probably be sent over TCP.

Answer 3

I don't recommend using TCP for reliable transport in games - the latency is too high, and the overhead of the transport will chew up your bandwidth. Instead you need to implement a reliable messaging system on top of UDP that uses ACKs to handle the sending and resending of messages. Consider the following situation - two machines are sending state information back and forth.

Machine A on Seq#100 creates object. In the packet it sends out, labelled #100, it includes the message.

Machine B receives the packet, and respond with the ACK of seq# 100. It creates the object.

Machine A receives the ACK of 100.

Later, Machine A destroys the object on seq# 200 and the same thing happens. A packet goes out with seq#200, Machine B acks it, and Machine A receives the ack.

This is great but how does it help us? Well if Machine B never receives seq#100, it can't ACK it, so machine A continues to send all the messages from previous sequence numbers until the ack is received - so seq# 101 has the create in it, seq#102 has the create in it, etc, until the ACK is received. In fact, because of latency, seq#101 is likely created and sent before the ack for 100 could ever be received, which helps decrease latency.

Should the ACK from Machine B to machine A be lost, then Machine A keeps sending the message until an ACK is received - it can't assume it has been received so it has no other choice but to do so.

Now if there is such dramatic packet loss that nothing is ack'ed since #100 when the object is destroyed, then the outgoing packet will contain both the create AND the destroy data, which is perfect. The messages should be put into the packet in sequence number order, so they maintain their timeline.

The key here is to put your messages into a queue, sorted by time, and to send the messages out until they are acknowledged. By having them in a queue, you add new messages to the back, and prune old messages from the front. Adding them is easy because they are always sorted in the queue. Further, the ACK sent from machine B to machine A should be bundled with whatever data that machine B needs to send back - there is no point to make a distinct packet - that's a waste of bandwidth.

Note that this is only for reliable messages. Unreliable messages work into this system just fine but are only sent on the seq# they are generated- they are not held onto and resent for multiple frames.

What I like about this system is that your create/destroy messages are kept in sync with the packet data for position, health, etc. which would be included in the same packet.

In a peer to peer game each peer would send the same data, so the packet layout might look like this:

Sequence # Acked Sequence # Number of messages in the packet each message additional payload data such as delta compressed snapshot data

Packet size grows as latency increases, but there are some ways to help mitigate that, such as slowing down the frequency you send packets at. I think Gaffer talks about that strategy.

I hope some of this makes sense.