There are things like Protocol Buffers, MessagePack, Capn Proto, Flatbuffers, etc. that can handle serialization for you, but I ended up doing something relatively simple. I wanted something that worked on my own data structures instead of switching to someone else's, and I didn't need most of the features they offered (backwards/forwards compatibility, optional fields, etc.). Backwards/forwards compatibility is very useful for serializing to disk or database, but for serializing to network, I didn't need it.
I did something structured like operator << (ostream&, T&)
, operator >> (ostream&, T)
, but for binary instead of the standard stream formatting. I don't modify the original data structures at all. Boost calls this “non-intrusive” serialization.
Let's say you have a Point
type with x
, y
, _temporary
, but you don't want _temporary
to be serialized. With operator <<
you could write it like this:
ostream& operator << (ostream& out, const Point& p)
{
return out << p.x << p.y;
}
The main idea is that <<
is already defined on numbers, so I'm defining the new operation <<
on Point
in terms of the existing operations on numbers.
I took that same idea but made it work for binary and json serialization.
If I have serialize(int)
, serialize(vector<T>)
, etc., then for a user defined type like Point
, I only need to define a function:
serialize(const Point& p)
{
serialize(p.x);
serialize(p.y);
}
In that function I don't mention _temporary
so it doesn't get serialized. If you then want to write out a LineSegment
and it's made up of two Points
, and you've already defined serialize(Point)
you can define serialize(LineSegment)
by calling serialize on the two points.
I also wanted to not have to write that once per type of operation (binary serialize, binary deserialize, json serialize, json deserialize, imgui gui output, etc.) so I abstracted a bit more, to something like this:
visit(Operation& op, const Point& p)
{
visit(op, p.x);
visit(op, p.y);
}
I also wanted to handle messages of varying types. Instead of writing out the type for every single struct, I use mapbox variant, and use that type only in places where the type can vary. To write out a variant type, I write out the variant index, then I write out the data. To read in a variant type, I read in the variant index, then construct a struct of the corresponding type, then read in the data. It became much simpler this way.
So all of this is a high level description but it might help to see the actual code with some tests. I'm not going to say my code is great, but here it is: link. Take a look at traverse.h to see how BinarySerialize works. There's a function to write out a signed int, an unsigned int, a string, and a vector. Then if you write out a Point
, it writes out the x
and the y
, which end up recursively calling the functions to write out whatever type those fields are. For integers, I originally just wrote out the bytes, but switched to Google's zig-zag format because it's more compact, handles little/big endian, and handles 32bit/64bit. I haven't yet uploaded the mapbox variant serialization but if you need that let me know.
Other approaches that might be worth a look: boost fusion, this blog post, the cereal library, self-aware structs. For various reasons I ended up not using any of these and made my own :( but I'm happy that my code is rather short and simple, and can also "serialize" to a gui for in-game visualization or to std::cout for logging.