There are things like Protocol Buffers, MessagePack, Capn Proto, Flatbuffers, etc. that can handle serialization for you, but I ended up doing something relatively simple. I wanted something that worked on my own data structures instead of switching to someone else's, and I didn't need most of the features they offered (backwards/forwards compatibility, optional fields, etc.). Backwards/forwards compatibility is very useful for serializing to disk or database, but for serializing to network, I didn't need it.
I did something structured like
operator << (ostream&, T&),
operator >> (ostream&, T), but for binary instead of the standard stream formatting. I don't modify the original data structures at all. Boost calls this “non-intrusive” serialization.
Let's say you have a
Point type with
_temporary, but you don't want
_temporary to be serialized. With
operator << you could write it like this:
ostream& operator << (ostream& out, const Point& p)
return out << p.x << p.y;
The main idea is that
<< is already defined on numbers, so I'm defining the new operation
Point in terms of the existing operations on numbers.
I took that same idea but made it work for binary and json serialization.
If I have
serialize(vector<T>), etc., then for a user defined type like
Point, I only need to define a function:
serialize(const Point& p)
In that function I don't mention
_temporary so it doesn't get serialized. If you then want to write out a
LineSegment and it's made up of two
Points, and you've already defined
serialize(Point) you can define
serialize(LineSegment) by calling serialize on the two points.
I also wanted to not have to write that once per type of operation (binary serialize, binary deserialize, json serialize, json deserialize, imgui gui output, etc.) so I abstracted a bit more, to something like this:
visit(Operation& op, const Point& p)
I also wanted to handle messages of varying types. Instead of writing out the type for every single struct, I use mapbox variant, and use that type only in places where the type can vary. To write out a variant type, I write out the variant index, then I write out the data. To read in a variant type, I read in the variant index, then construct a struct of the corresponding type, then read in the data. It became much simpler this way.
So all of this is a high level description but it might help to see the actual code with some tests. I'm not going to say my code is great, but here it is: link. Take a look at traverse.h to see how BinarySerialize works. There's a function to write out a signed int, an unsigned int, a string, and a vector. Then if you write out a
Point, it writes out the
x and the
y, which end up recursively calling the functions to write out whatever type those fields are. For integers, I originally just wrote out the bytes, but switched to Google's zig-zag format because it's more compact, handles little/big endian, and handles 32bit/64bit. I haven't yet uploaded the mapbox variant serialization but if you need that let me know.
Other approaches that might be worth a look: boost fusion, this blog post, the cereal library, self-aware structs. For various reasons I ended up not using any of these and made my own :( but I'm happy that my code is rather short and simple, and can also "serialize" to a gui for in-game visualization or to std::cout for logging.