# What is better? Lots of small TCP packets, or one long one? [closed]

I am sending quite a bit of data to and from a server, for a game I'm making.

I currently send location data like this:

sendToClient((("UID:" + cl.uid +";x:" + cl.x)));
sendToClient((("UID:" + cl.uid +";y:" + cl.y)));
sendToClient((("UID:" + cl.uid +";z:" + cl.z)));


Obviously it is sending the respective X, Y, and Z values.

Would it be more efficient to send data like this?

sendToClient((("UID:" + cl.uid +"|" + cl.x + "|" + cl.y + "|" + cl.z)));

• Packet loss is usually under 5%, in my limited experience. – mucaho Mar 19 '15 at 21:10
• Does sendToClient actually send a packet? If so, how did you make it do that? – user253751 Mar 20 '15 at 1:19
• @mucaho I've never measured it myself or anything, but I'm surprised TCP is that rough around the edges. I would've hoped for something more like 0.5% or less. – Panzercrisis Mar 20 '15 at 12:46
• @Panzercrisis I must agree with you. I personally feel that a 5% loss would be unacceptable. If you think about something like me sending, say, a new ship spawned in the game, even a 1% chance of that packet not being received would be disastrous, because I would get invisible ships. – joehot200 Mar 20 '15 at 15:29
• don't freak out guys, I meant the 5% as an upper bound :) in reality it's much better, as noted by other comments. – mucaho Mar 20 '15 at 19:42

A TCP segment has quite a lot of overhead. When you send a 10 byte message with one TCP packet you actually send:

• 16 bytes of IPv4 header (will increase to 40 byte when IPv6 becomes common)
• 16 bytes of TCP header

resulting in 42 bytes of traffic for transporting 10 byte of data. So you only make use of less than 25% of your available bandwidth. And that does not yet account for the overhead which the lower-level protocols like Ethernet or PPPoE consume (but these are hard to estimate because there are so many alternatives).

Also, many small packets put more strain on routers, firewalls, switches and other network infrastructure equipment, so when you, your service provider and your users don't invest in high-quality hardware, this might turn into another bottleneck.

For that reason you should try to send all data you have available at once in one TCP segment.

Regarding handling packet loss: When you use TCP you don't need to worry about that. The protocol itself ensures that any lost packets are resent and packets are processed in order, so you can assume that all packets you send will arrive at the other side, and they will arrive in the order you send them. The price for this is that when there is packet loss, your player will experience considerable lag, because one dropped packet will halt the whole data-stream until it was re-requested and received.

When this is a problem, you can always use UDP. But then you need to find your own solution for lost and out-of-order messages (it at least guarantees that the messages which do arrive, arrive complete and undamaged).

• Does the overhead from the way TCP recovers packet loss vary worth anything depending on the packets' sizes? – Panzercrisis Mar 19 '15 at 19:24
• @Panzercrisis Only in so far as there is a larger packet which needs to be resent. – Philipp Mar 19 '15 at 19:40
• I would note that the OS will almost certainly apply Nagles Algorithm en.wikipedia.org/wiki/Nagle%27s_algorithm to outgoing data, this means that it does not matter if in the app you do separate writes or combine them, it will combine them before actually passing them out via TCP. – Vality Mar 19 '15 at 23:44
• @Vality Most socket APIs I've used allow to activate or deactivate nagle for each socket. For most games I would recommend to deactivate it, because low latency is usually more important than conserving bandwidth. – Philipp Mar 20 '15 at 8:14
• Nagle's algorithm is one, but not the only reason data may be buffered on the sending side. There is no way to reliably force data sends. Also, buffering/fragmentation may occur anywhere after data is sent, either at NATs, routers, proxies, or even the receiving side. TCP does not make any guarantees regarding the size and timing at which you receive data, only that it will arrive in order and reliably. If you need size guarantees, use UDP. The fact that TCP seems easier to understand doesn't make it the best tool for all problems! – Panda Pajama Mar 20 '15 at 10:32

One large one (within reason) is better.

As you said, packet loss is the main reason. Packets are generally sent in frames of a fixed size, so it is better to take up one frame with a big message than 10 frames with 10 small ones.

However with standard TCP, this isn't really an issue, unless you disable it. (It is called Nagle's algorithm, and for games you should disable it.) TCP will wait for a fixed timeout or until the package is "full". Where "full" is would be some slightly magic number, determined in part by the frame size.

• I've heard of Nagle's algorithm, but is it really a good idea to disable it? I just came right from a StackOverflow answer where someone said that it is more efficient (and for obvious reasons I want efficiency). – joehot200 Mar 19 '15 at 16:46
• @joehot200 The only correct answer to that is "it depends". It is more efficient for sending lots of data, yes, but not for real-time streaming that games tend to need. – D-side Mar 19 '15 at 17:11
• @joehot200: Nagle's algorithm interacts poorly with a delayed-acknowledgment algorithm which certain TCP implementations sometimes use. Some TCP implementations will delay sending an ACK after they get some data if they expect more data to follow soon after (since acknowledging the later packet would implicitly acknowledge the earlier one as well). Nagle's algorithm says that a unit shouldn't send a partial packet if it has sent some data but not heard an acknowledgment. Sometimes the two approaches interact badly, with each parties waiting for the other to do something, until... – supercat Mar 19 '15 at 20:23
• ...a "sanity timer" kicks in and resolves the situation (on the order of a second). – supercat Mar 19 '15 at 20:24
• Unfortunately, disabling Nagle's algorithm will do nothing to prevent buffering on the other host side. Disabling Nagle's algorithm does not ensure you get one recv() call for each send() call, which is what most people are looking for. Using a protocol that guarantees this, like UDP does though. "When all you have is TCP, everything looks like a stream" – Panda Pajama Mar 20 '15 at 2:15

All the previous answers are incorrect. In practice, it doesn't matter whether you issue one long send() call or several small send() calls.

As Phillip states, a TCP segment has some overhead, but as an application programmer, you have no control over how segments are generated. In simple terms:

One send() call does not necessarily translate to one TCP segment.

The OS is completely free to buffer all of your data and send it in one segment, or take the long one and break it into several small segments.

This has several implications, but the most important one is that:

One send() call, or one TCP segment does not necessarily translate to one successful recv() call on the other end

The reasoning behind this is that TCP is a stream protocol. TCP treats your data as a long stream of bytes, and has absolutely no concept of "packets". With send() you add bytes to that stream, and with recv() you get bytes off the other side. TCP will aggressively buffer and split your data wherever it sees fit to make sure your data gets to the other side as fast as possible.

If you want to send and receive "packets" with TCP, you have to implement packet start markers, length markers and so on. How about using a message oriented protocol like UDP instead? UDP guarantees one send() call translates to one sent datagram and to one recv() call!

When all you have is TCP, everything looks like a stream

• Prefixing each message with a message length is not so tricky. – ysdx Mar 20 '15 at 8:21
• You have one switch to flip when it comes to packet aggregation, whether Nagle's Algorithm is in effect or not. It's not uncommon that it's off in game networking to ensure prompt delivery of underfilled packets. – Lars Viklund Mar 20 '15 at 8:45
• This is completely OS or even library specific. Also you DO have a lot of control - if you want. It is true you do not have total control, TCP is always allowed to combine two messages, or split one if it doesn't fit the MTU, but you can still hint it in the right direction. Setting various config settings, manual sending messages 1 second apart or buffering data and send it in one shot. – Dorus Mar 20 '15 at 9:05
• @ysdx: no, not on the sending side, but yes on the receiving side. Since you have no guarantees on where exactly you will get data on recv(), you need to make your own buffering to compensate for that. I would rank it in the same difficulty as implementing reliability over UDP. – Panda Pajama Mar 20 '15 at 10:11
• @Pagnda Pajama: A naive implementation of the receiving side is: while(1) { uint16_t size; read(sock, &size, sizeof(size)); size = ntoh(size); char message[size]; read(sock, buffer, size); handleMessage(message); } (omitting error handling and partial reads for brevity but it does not change much). Doing this is a select does not add much more complexity and if you are using TCP you probably need to buffer the partial messages anyway. Implementing robust reliability over UDP is much more complicated than that. – ysdx Mar 20 '15 at 10:27

Many small packages is fine. In fact, if you are worried about TCP overhead, just insert a bufferstream that collect up to 1500 chars (or whatever your TCP MTUs is, best to request it dynamically), and deal with the problem in one place. Doing so spares you the overhead of ~40 bytes for every extra package you would otherwise have created.

That said, it's still better to send less data, and building larger objects help there. Ofcourse it's smaller to send "UID:10|1|2|3 than to send UID:10;x:1UID:10;y:2UID:10;z:3. In fact, also at this point you should not be reinventing the wheel, use a library like protobuf that can decrease data like that to a 10 byte string or less.

Only thing you should not forget is to insert a Flush commands on your stream at relevant locations, because as soon as you stop adding data to your stream, it might wait infinite before it send anything. Really problematic when your client is waiting for that data, and your sever wont send anything new until the client send the next command.

Package loss is something you can affect here, marginally. Every byte you send can potentially be corrupted, and TCP will automatically request a retransmission. Smaller packages means a lower chance for every single package to be corrupted, but because they add up on the overhead, you send even more bytes, increasing the odds of a lost package even more. When a package is lost, TCP will buffer all succeeding data until the missing package is resend and received. This result in a large delay (ping). While the total loss in bandwidth because of package loss might be negligible, the higher ping would be undesirable for games.

Bottom line: Send as little data as possible, send large packages, and do not write your own low level methods to do so, but rely on well known libraries and methods like bufferstream and protobuf to handle the heavy lifting.

• Actually for simple things like this its easy to roll your own. Much easier than going through a 50 page documentation to use another person's library, and then after that you've still got to deal with their bugs and gotchas. – Pacerier Mar 20 '15 at 9:18
• True, writing your own bufferstream is trivial, that's why I called it a method. You still want to handle it in one place, and not integrate your buffer logic with your message code. As for Object Serialization, I highly doubt you get something better than the thousands of man-hours others put in there, even if you DO try, I strongly suggest you benchmark your solution against known implementations. – Dorus Mar 20 '15 at 9:26

Although being a neophyte to network programming myself, I would like to share my limited experience by adding a few points:

• TCP does imply an overhead - you must measure the relevant statistics
• UDP is the de facto solution for networked gaming scenarios, but all implementations that rely on it have an extra, CPU-side algorithm to account for packets being lost or being sent out of order

Concerning measurements, the metrics that should be considered are:

As mentioned, if you find out that you're not limited in a sense and could use UDP, go for that. There are some UDP-based implementations out there, so you don't have to reinvent the wheel or work against years of experience and proven experience. Such implementations worth mentioning are:

Conclusion: since a UDP implementation could outperform (by a factor of 3x) a TCP one, it makes sense to consider it, once you've identified your scenario to be UDP friendly. Be warned! Implementing the full TCP stack on top of UDP is always a bad idea.

• I was using UDP. I only just switched to TCP. UDP's packet loss was simply unacceptable for crucial data that the client needed. I may send movement data via UDP. – joehot200 Mar 20 '15 at 15:30
• So, the best things that you can do are: indeed, use TCP just for crucial operations OR use an UDP based software protocol implementation (with Enet being simple and UDT being well tested). But first, measure the loss and decide whether UDT would bring you an edge. – teodron Mar 20 '15 at 16:36