Is state change command less reliable than sending the entire state?

Question

I'm asking for a game that I develop right now. The game is using TCP communication, there's server and multiple clients. While players in a room are playing the game, I send state update commands in a custom binary protocol every 33 ms. (30 FPS) Problem is those states are huge - there are players moving, objects spawning and also destructible blocks.

If I were to send all this data a single state update to every player will be huge, resulting in huge bandwidth.

So I'm thinking different approach:

1. Initially send the huge state with all moving and stationary objects, when the room starts
2. Do the state update commands every 33 ms, but include just the moving objects, and not include the state and location of the stationary objects.
3. For stationary objects, issue separate commands to tell they are destroyed or spawned and expect clients to update their copy of the state.

My concern is that it may somehow happen that due to not including the stationary objects in every state update, clients may somehow miss the separate commands that update on those and not reflect that change in their copy of the state.

Here is a visual representation of what I imagine:

Less bandwidth (is it less reliable?):

game started -> obj1 {id,location}, obj2{id,location}, obj3{id,location}  
state update 1 -> obj1 {id,location}, obj2{id,location} #assumes obj3 is there  
state update 2 -> obj1 {id,location}, obj2{id,location} #assumes obj3 is there  
state update 3 -> obj1 {id,location}, obj2{id,location} #assumes obj3 is there  
object removed -> obj3 #removing the object locally  
state update 4 -> obj1 {id,location}, obj2{id,location} #obj3 not present at this point

More bandwidth - include all objects in every state update and assume if something is not present, it has been removed (is it more reliable?):

game started -> obj1 {id,location}, obj2{id,location}, obj3{id,location}  
state update 1 -> obj1 {id,location}, obj2{id,location}, obj3{id,location}  
state update 2 -> obj1 {id,location}, obj2{id,location}, obj3{id,location}  
state update 3 -> obj1 {id,location}, obj2{id,location} #obj3 is not here, so removing it locally  
state update 4 -> obj1 {id,location}, obj2{id,location} #obj3 not present at this point

Answer 1

I guess most will agree that one should (basically) never re-send information that was already sent. Distributing only the changes produces a correct end result, provided the changes went through successfully. Obviously :-). It's a matter of the greed every game coder must have. Never ever do anything unnecessary.

As you know, TCP is problematic in the sense that it struggles for an eternity to perform well (and btw adds latency for the sake of being a good network citizen). But things change rapidly inside a game. Nobody cares about old, outdated data, it is a much worth as no data. Maybe consider UDP, or a hybrid of TCP and UDP?

I understand your concern, everyone does and everyone knows it. What to do with the lag, network errors, high ping bastards? 95% got the Go signal, how to deal with that 5 %?

Depends on your game, goals, business, whatnot. Apparently, one wants to give them an as good experience as possible?

Hence, you must make the system tolerant. Ofc, one could just kick out the erratic cases, but that would be a bit brutal. So one wants to keep it all alive, even if something's wrong.

An object should be allowed to become "unstable", "unreliable". But how that affects the game logic, is a thing only you know.

Imho there is no simple answer to your question. Basically:

• consider (and read on) the TCP. It delivers, no matter what. But are you willing to wait, if there are problems? Are you willing to make a deal with TCP, as your only delivery service?
• send changes only (at least mostly; naturally, a full set occasionally is not at all forbidden). Pre-calc, cache, sparsify, persist. Maybe let the actual change event raise and time a transmission?
• can always transmit tiny checksums to pack/inform a snapshot of multiple states. Endpoints (server, peer) can calc them individually, locally; compare with the checksum they got in and react on a mismatch => something has changed (after which a peer can ask for an update, etc., to fix the problem).
• have the game objects be "impatient", "self-aware". Meaning they count their time and timeout, to ask for new data, if they didn't get any attention for some time.
• "grey out" untrustworthy objects/states, both visually and logically (meaning in the maths). This is where the tolerance comes in...

Never forget rule #1: Add fog

(Ie: Hide the problems. Not literally by using fog, but by using whatever smart actions available)

---

Edit, in reply to the comment.

Logically, TCP guarantees delivery, untill a timeout that would best be called "long hang". Additionally, TCP delivers in the correct order. Hence, the logical chain of events will not be broken (but the timing between peers may cause issues, see below). Nevertheless, TCP allows you to distribute changes only, in a safe manner. The communication in itself maintains the validity checking, and the assumption is that the computer receiving the data cannot calculate erratically if received correct data -> no further errors will occur.

Timing issues: I guess people who consider you case think further, as i did too. There are "what if"-questions related to the overall functionality. But if leaving them on the discussion side only, the answer to your sole question is that sending changes only is as safe as "it can be". In addition, the fact that less data was sent, in fact increases the overall performance, reducing chances for problems.

Answer 2

But are there cases where the client and server states become different? (for instance lost command or something else).

If you are using floating point without controlling every detail tightly / correctly, then yes, you may well need regular re-syncing. Floating point primarily creates problems in respect to rounding and order of arithmetic operations. (Related to incorrect ordering of operations and thus also potentially causing unwanted effects in systems built on floating point arithmetic, are non-deterministic threading solutions like OpenMP; c.f. libDOMP thesis). IEEE-754 floating point can be (according to most sources, note the varied answers here) deterministic, but it's generally how you work with it that can cause serious issues; see various game dev articles around the web on float determinism. See libfixmath which may help you avoid the usual implications of IEEE-754 floating point.

About TCP

Really your question perhaps should be, "Can TCP ever fail to deliver packets?" I can only repeat what another here has already said: TCP is reliable, that is, it will not allow any messages to go undelivered (UDP, OTOH, will allow that). So if there's a failure, it's not on TCP's part, but on the part of your code. TCP waits (indefinitely, potentially) till it receives the appropriate packet in a sequence. No matter how many packets are lost on the wire, TCP (unless managed otherwise) will keep on waiting till the user-specified timeout is reached (usually on the order of several minutes).

Further to your question

It's perfectly normal to do state catch up every so often, even with deltas, if there is any chance that your application doesn't catch all cases you would hope. You can even have players set the frequency (every n minutes) in the lobby, but ultimately the more frequently you do it, the less likely you are to experience a checksum mismatch / sync fault, so you may want to fix the frequency accordingly.

You should pay more attention to building systems through which all traffic is routed, and which thus act as a basis for comparison so that you are able to catch, in retrospect, exactly what went wrong, and when. Determinism, logging and diffing, are essential aspects of distributed development. So in your particular use case, you can own the problem and the solution by knowing what is happening at all times and being able to track a problem down in the very frame it goes wrong.

You may also want to look into the benefits that peer-to-peer lockstep brings in this regard. And no, if your simulation is built correctly, you should never need to update immovable objects unless they spawn or despawn. (The same actually applies to moving objects, as spawn / despawn is a separate class of thing from motion / firing.)

Re your proposed message approach

For Peer-to-Peer, if you simply leave certain entities out, what if the receiver has reason to believe those entities still exist, but the sender does not? This immediately becomes a point of contention. I suppose it could be made to work, but I think it is more important to treat all your networked communications as a concept of state changes or deltas rather than sending full state while assuming that if certain things are missing from that state, they should naturally be destroyed by other clients then and there. When we talk about spawn/despawn for example, that there is a delta.

For authoritative client-server, your approach makes sense. However, for the "fire-and-forget" style of comms you are suggesting, a connectionless protocol like UDP may make more sense than TCP.