11
\$\begingroup\$

We are using JSON right now and want to move to a binary format for some types of messages between client and server.

Should I just read structs into the socket? Use proticol buffers/thrift?

How should I represent arrays of data?

What should the interface look like for packing/unpacking data?

Improve this question
\$\endgroup\$

2 Answers 2

Reset to default
12
\$\begingroup\$

Assuming...

  1. you are talking about converting to a buffer of bytes
  2. You are using UDP and performance is a concern

Try to avoid wasting space in your packet for defining structure. I.E. send, at minimum, a byte to denote the type of packet, then just assume each packet received follows the predefined structure for that type of packet

Should I just read structs into the socket? Use proticol buffers/thrift?

  • Yes, read the whole struct IF you NEED the whole struct
  • No, make the packet structure yourself, This will surely be smaller than serialization using these methods; you should know exactly what data the packet should include

How should I represent arrays of data?

  • As arrays of data. When receiving continue reading the buffer until end of data to avoid sending a Count of the array's elements

What should the interface look like for packing/unpacking data?

  • You could easily setup a bunch of methods to convert basic types to bytes, from there build on these methods to convert custom types as well. The specifics on how to do this could be found almost anywhere I'm sure (I use C# personally)

One last thing, packet size is a concern, especially for a snapshot: size = packetSize x entities x connectedPlayers; So you might have 60 x 10 x 16 = 9,600 bytes per packet Then sending this 20 times a second: = 192,000 bps = 187 KBps. This is obviously a high bandwith upload speed. Thus the need to minimize each of the factors contributing to packet size where possible.

This article has helped me immensely: Valve Multiplayer Networking

Improve this answer
\$\endgroup\$
4
  • \$\begingroup\$ Another article that I discovered while reading various object serialisation and networking questions here a few weeks ago was this one which describes how the Unreal engine does it. A good point of comparison for the Valve source. \$\endgroup\$
    – Martin Foot
    Aug 31, 2011 at 8:42
  • 1
    \$\begingroup\$ Your array method is not going to work in the general case - where is the 'end of data'? Even if your messages are delimited, it would mean you can't have more than 1 array per struct. To fix this, the original poster could either stick to fixed length arrays, or ensure there's only 1 array per struct (at the end of the struct), or send a count value at the start of the array. \$\endgroup\$
    – Kylotan
    Aug 31, 2011 at 10:48
  • \$\begingroup\$ Just one more tip: Remember to treat endianess, this can be very anoying if you don't know that such thing exist. \$\endgroup\$
    – user9471
    Aug 31, 2011 at 13:03
  • \$\begingroup\$ Good point @Kylotan , agree that in certain cases this extra data can't be avoided; but if I find myself adding multiple arrays to a single packet, I would consider sending multiple packets instead \$\endgroup\$
    – indeed005
    Sep 1, 2011 at 0:47
1
\$\begingroup\$

This problem has been solved by Google and Facebook:

  1. Google's Protocol Buffers — Google is a big user of C++:

    Protocol Buffers are a way of encoding structured data in an efficient yet extensible format. Google uses Protocol Buffers for almost all of its internal RPC protocols and file formats.

  2. Apache Thrift (formerly by Facebook):

    Thrift is a software framework for scalable cross-language services development. It combines a software stack with a code generation engine to build services that work efficiently and seamlessly between C++, Java, Python, PHP, Ruby, Erlang, Perl, Haskell, C#, Cocoa, JavaScript, Node.js, Smalltalk, and OCaml.

Improve this answer
\$\endgroup\$
4
  • \$\begingroup\$ Google's Protocol buffers are too slow for larger scale, real time games. I did find them quite nice for prototyping and smaller player numbers because of the versioning, however. As usual, your profiler will tell the real story. \$\endgroup\$
    – Patrick Hughes
    Sep 2, 2011 at 17:48
  • \$\begingroup\$ Well, they're good enough for Google, and Google scales pretty well, and they worked well when I used them. That's why I recommended them. \$\endgroup\$
    – a paid nerd
    Sep 2, 2011 at 20:57
  • \$\begingroup\$ Google doesn't require real time performance. Google does require reliability and up-time, both of which are well served by Protocol Buffers. The complexity of all that fallback versioning and boilerplate code generation adds overhead and when you're sending and receiving 1000 updates at 50-100msec intervals it adds up. Profile a Protocol buffer several versions old against a coded serializer specific to the data at hand. @indeed005 has the gist of it. \$\endgroup\$
    – Patrick Hughes
    Sep 2, 2011 at 21:22
  • \$\begingroup\$ +1, because although these formats are a bit too large and slow for most real-time or high-bandwidth games (due to containing extra information that allows you to reconstruct arbitrarily complex packets), that's not to say they aren't useful in some games, eg. turn-based ones. If optimising every resource isn't necessary then these formats can save you a lot of time, and they're certainly more efficient than JSON. \$\endgroup\$
    – Kylotan
    Sep 4, 2011 at 12:27

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .