Example/Tutorial of passing structured data over network C++

Question

This is very general question, about strategies how to pack same information about state of my game-world on server ( e.g. position/orientation of players, projectiles, etc. ) into data stream so that I can sent them over network and unpack them on the client side to draw on screen.

I know it's called serialization. But still I have many doubts.

1. My game already has some structure of objects. I do not want to re-design it ( too much ) in order to use some automatic serialization. I also do not want to make dependency on an other library (such as boost::serialize ) if not necessary
2. My classes/objects ^{e.g.(class Airplane)} contain some properties which should be passed over network to the client ^{( e.g. position and orientation )} but also some which should not ^{( e.g. some temporary variables relevant just for computations on server; pointers to nearest neighbor objects; array of Lift and Drag coefficient for set of angles-of-attack )}
3. If I pack more objects into one message/packet I will probably need some protocol and parser to unpack the objects properly on the other side. The client should know of which Class type data are and to which instance it should be assigned.

Probably by trial and error I can figure it by myself, but it would take some time, and it would be helpful to have some example solution of similar problem to start with.

---

background - I have some skill with physics and graphics programming (Java,C++,Frotran,GLSL,OpenCL...), but I know absolutely nothing about networking. I have done some games into which I would like multiplayer. Just simple games with only few players and moving objects (projectiles). For reference consider something like LIERO

Answer 1

There are things like Protocol Buffers, MessagePack, Capn Proto, Flatbuffers, etc. that can handle serialization for you, but I ended up doing something relatively simple. I wanted something that worked on my own data structures instead of switching to someone else's, and I didn't need most of the features they offered (backwards/forwards compatibility, optional fields, etc.). Backwards/forwards compatibility is very useful for serializing to disk or database, but for serializing to network, I didn't need it.

I did something structured like operator << (ostream&, T&), operator >> (ostream&, T), but for binary instead of the standard stream formatting. I don't modify the original data structures at all. Boost calls this “non-intrusive” serialization.

Let's say you have a Point type with x, y, _temporary, but you don't want _temporary to be serialized. With operator << you could write it like this:

ostream& operator << (ostream& out, const Point& p)
{
    return out << p.x << p.y;
}

The main idea is that << is already defined on numbers, so I'm defining the new operation << on Point in terms of the existing operations on numbers.

I took that same idea but made it work for binary and json serialization.

If I have serialize(int), serialize(vector<T>), etc., then for a user defined type like Point, I only need to define a function:

serialize(const Point& p)
{
    serialize(p.x);
    serialize(p.y);
}

In that function I don't mention _temporary so it doesn't get serialized. If you then want to write out a LineSegment and it's made up of two Points, and you've already defined serialize(Point) you can define serialize(LineSegment) by calling serialize on the two points.

I also wanted to not have to write that once per type of operation (binary serialize, binary deserialize, json serialize, json deserialize, imgui gui output, etc.) so I abstracted a bit more, to something like this:

visit(Operation& op, const Point& p)
{
    visit(op, p.x);
    visit(op, p.y);
}

I also wanted to handle messages of varying types. Instead of writing out the type for every single struct, I use mapbox variant, and use that type only in places where the type can vary. To write out a variant type, I write out the variant index, then I write out the data. To read in a variant type, I read in the variant index, then construct a struct of the corresponding type, then read in the data. It became much simpler this way.

So all of this is a high level description but it might help to see the actual code with some tests. I'm not going to say my code is great, but here it is: link. Take a look at traverse.h to see how BinarySerialize works. There's a function to write out a signed int, an unsigned int, a string, and a vector. Then if you write out a Point, it writes out the x and the y, which end up recursively calling the functions to write out whatever type those fields are. For integers, I originally just wrote out the bytes, but switched to Google's zig-zag format because it's more compact, handles little/big endian, and handles 32bit/64bit. I haven't yet uploaded the mapbox variant serialization but if you need that let me know.

Other approaches that might be worth a look: boost fusion, this blog post, the cereal library, self-aware structs. For various reasons I ended up not using any of these and made my own :( but I'm happy that my code is rather short and simple, and can also "serialize" to a gui for in-game visualization or to std::cout for logging.

Answer 2

1 & 2) I would suggest the easiest approach without adding too much overhead would be to add seperate replication logic for networked objects. A Replicate and an OnReplicated method to each network enabled object (perhaps via an interface or abstract base class). The Replicate function would take an ostream object and use the operators to add the data needed (don't forget to encode std::string as spaces make the read from the istream after passing it from the network act poorly). Then you would add some kind of networkId for the object's instance so that when it is pushed across the wire, the server and other clients would know which object to grab a pointer to and call OnReplicated, passing in an istream object to read the data from. Here is some example code (where UUID is a struct, or value that is guaranteed unique):

class BaseNetworkReplicable
{    
public:
    BaseNetworkReplicable();
    virtual ~BaseNetworkReplicable();
    virtual void Replicate(std::ostream& out) const;
    virtual void OnReplicated(std::istream& in);

private:
    UIID networkUIID;
};

And then the implementation (where MyUIIDGenerator is some kind of class that guarantees a unique value every time its MakeUIID method is called):

BaseNetworkReplicable::BaseNetworkReplicate()
{
    networkUIID = MyUIIDGenerator::MakeUIID();
}

BaseNetworkReplicable::~BaseNetworkReplicate()
{
}

void BaseNetworkReplicable::Replicate(std::ostream& out) const
{
    out << networkUIID;
}

void BaseNetworkReplicable::OnReplicated(std::istream& in)
{
    in >> networkUIID;
}

Finally, you would need to define a network replication protocol. Could be something as simple as this definition:

struct ReplicationPacket
{
    static const int ReplicationPacketTypeId;
    int networkUIID;
    int length;
    char* data; // or a stream
}

Then when you write it, you simple write the networkid, the length and the data to the wire. From the receiving end you ask a lookup table for a pointer to the BaseNetworkReplicable object with that networkUUID that matches the networkUUID sent in the ReplicationPacket.

By having a base class type of BaseNetworkReplicable, you don't need the extra logic of knowing the class type. This is essential for serialization, but since you are using virtual functions, it should auto-magically deserialize the replicated data for you.

3) I mentioned above that you really don't need to know the class type anymore if you use a seperate Replication pipeline. However, you mentioned packing multiple objects. You would just create a seperate ReplicationPacketCollection:

struct ReplicationPacketCollection 
{
    const int replicationPacketCollectionTypeId;
    int packetCount;
    int totalPayloadLength; //sum of all ReplicationPacket lengths + ((3 * sizeof(int)) * number of packets) + (2 * sizeof(int))
    ReplicationPacket* collection;    
}

I have covered a lot of the base objects needed to start up the framework for network replication but there is a lot more to cover. Such other considerations are:

• Who owns the object that controls the true state
• How to tell connected 'slave devices' that a new object was created on the 'master device' and need to add an entry for that network object in the replication lookup table
• Registering replicated objects
• the network code to write to and read from the network stream
• Handling pointers (I solved this by writing a NetworkUIID instead of a pointer and has some helper pointer serialization logic to fetch the correct pointer on the devices on the other side of the network.)

I hope this helps you get started!