How can I handle versioning changes when saving assets?

Question

I've been working on an RPG now for a while and I use two of different serialization techniques.

• Enemies, Weapons, items are as saved as XML.
• Maps and events are saved as "controlled binary" (every class get a save/load method and they decide what they want to save/load).

But I have started to question my choice for maps and events. My concerns:

• I have created a map editor but I still miss being able to change small things by just opening the file.
• Changes mess so much up. Say that I want to add a variable to a class, if I don't load/save every map again it break later.

The first concern is hard to get around without changing my technique. I thought about changing to JSON, but it's a lot of work. I also think it look pretty ugly with [DataContract] and [DataMember] attributes everywhere.

That leave me with my second concern and I wonder how I can deal with it? Do you create a small program that loop through all maps and re-save them with the new variable? Because I start to get a couple of maps now and I still do it manually. It make me think twice everytime I want to do some changes as it create a lot of extra work.

Answer 1

There are lots of ways to handle the versioning problem; you can do it by having one load function per version, you can try to automate process by describing (via attributes usually) the transformation of the asset structure over time, you can do version-specific checks inside the load/save functions, et cetera.

I like the "describe the changes" approach but find that trying to do it via attributes gets awkward fast. I'd use functions instead; implement a function that transforms data in version N to data in version N + 1 for all your appropriate version. On load, check the version against the latest and if it isn't, run the data through all the appropriate versioning functions. Always save out the latest version.

This works best if you do the transformation when the data is still in a runtime key-value form. This means you will probably want to implement a representation for your data that is a "runtime bag of properties" approach, because you can't use the underlying key-value form of JSON or XML if you've got your own binary format. If you don't do this, you also may need to keep old class definitions around, which gets ugly. Being able to have your assets in this property bad format is also tremendously useful for game editor development.

During development as you iterate on your data it will naturally bubble up to the latest version and you can eventually delete the old versioning functions. This is more or less the same high-level approach that we used to version art assets (such as the maps) in Guild Wars 2.

---

Now, all that said, I think it's useful to support both text and binary serialization for assets. During development, keep all your data in a human-readable format based on XML or JSON. This can increase your iteration ability a lot because you don't need to build such complex tools around editing the data. You can return to being able to make simple quick tweaks by hand.

Second, assuming you even still want a binary format for shipping the game (which can improve file size or file IO times, so it's a valid desire), design your serialization and deserialization APIs to handle versioning. Versioning is still useful in a shipping context, because as some point you may want to ship updates or bug fixes. There are some documents describing the versioning abilities of .NET serialization and Boost's serialization that you may find interesting. If you are going to support both text and binary formats, make sure you test them occasionally (or build automated tests to do so, even better).

Answer 2

Use a markup language with attribute-value pairs like XML or JSON.

The parser can just ignore any attributes it doesn't understand or use defaults for any it doesn't find, which makes backward- and forward compatibility quite easy. Also, the format is human-readable so you can easily edit it with a text editor.

When you use an established language like XML or JSON you will also notice that many scripting languages support it, so when you still need to write a script to edit a large number of files, you will find it much easier to do.

The drawback of most of these languages is that they are quite verbose. That means the resulting files are much larger than they would need to be with an optimized binary format. Nowadays, file size doesn't matter too much in most situations. But in those where it does matter, the filesize can often be significantly reduced by compressing the file with a stock algorithm like zip.

Markup languages often don't allow random access unless the whole document is read from the hard drive and parsed. But in practice this doesn't matter that much, because hard drives are fastest with sequential reads. Randomly seeking multiple times to different parts of the same file can often be much slower than just reading the file in one go, even when it means that you read more data than you need to.

Answer 3

you can use protobuf. https://code.google.com/p/protobuf/ It gives you the advantages of json/xml, that you can easily extend it while beeing backwards compatible, plus the advantage of being binary. The workflow is, that you create a data format description in the protobuf language and then generate source code for serialization and deserialization. Source can be generated for several languages. Also it is a big advantage that you have a clear specification of your serialized data, in contrast to json where the specification is done implicitly in reading/writing.