There are lots of ways to handle the versioning problem; you can do it by having one load function per version, you can try to automate process by describing (via attributes usually) the transformation of the asset structure over time, you can do version-specific checks inside the load/save functions, et cetera.
I like the "describe the changes" approach but find that trying to do it via attributes gets awkward fast. I'd use functions instead; implement a function that transforms data in version
N to data in version
N + 1 for all your appropriate version. On load, check the version against the latest and if it isn't, run the data through all the appropriate versioning functions. Always save out the latest version.
This works best if you do the transformation when the data is still in a runtime key-value form. This means you will probably want to implement a representation for your data that is a "runtime bag of properties" approach, because you can't use the underlying key-value form of JSON or XML if you've got your own binary format. If you don't do this, you also may need to keep old class definitions around, which gets ugly. Being able to have your assets in this property bad format is also tremendously useful for game editor development.
During development as you iterate on your data it will naturally bubble up to the latest version and you can eventually delete the old versioning functions. This is more or less the same high-level approach that we used to version art assets (such as the maps) in Guild Wars 2.
Now, all that said, I think it's useful to support both text and binary serialization for assets. During development, keep all your data in a human-readable format based on XML or JSON. This can increase your iteration ability a lot because you don't need to build such complex tools around editing the data. You can return to being able to make simple quick tweaks by hand.
Second, assuming you even still want a binary format for shipping the game (which can improve file size or file IO times, so it's a valid desire), design your serialization and deserialization APIs to handle versioning. Versioning is still useful in a shipping context, because as some point you may want to ship updates or bug fixes. There are some documents describing the versioning abilities of .NET serialization and Boost's serialization that you may find interesting. If you are going to support both text and binary formats, make sure you test them occasionally (or build automated tests to do so, even better).