Game launcher / updater download delta diff

Question

I'm making a launcher for my game (using Unity3d).

I want it to have the following features :

Compare game version to latest version and update if necessary
Check files integrity and update if differences are found
Download/patch needed files

So comparing the game version is easy : I just parse a version.txt file and compare it's value to the latest server version.

Checking file integrity sounds easy too : Make a list of all files in the game folder, order the list, get a checksum and check if it corresponds.

Now here's the difficult part : Downloading the required files.

My first idea was : If a file checksum is different, download that file. But unity assets are compressed into large files, so even a simple change will lead to the download of a 2gb file...

So I'm now looking for a way do download only the different parts. I heard about VCDIFF / xdelta, apparently they can be useful to get a "delta patch" of different binaries. Cool. But how can I compare two folders if one of them is distant, without downloading it all before ?

Could torrent be an answer ? How are game/unity devs doing this ?

• Maybe rsync can be useful, but is it safe if hundreds of clients are simultaneously updating using rsync ?
• I heard about Unity's AssetBundles. Is that useful and is it ok to put ALL my assets in bundles ?
• How would I display a progress bar in unity showing the rsync progression ?

Answer 1

You need to start by separating the concept of updating from the concept of repairing.

The algorithm is roughly the following:

• Get the claimed version of the client.
• Ask the server for that version.
• The server returns the checksums for that version, and whatever or not there is a new version.
• Check the integrity of the files with the provided checksums.

Now, you have either a good or a corrupted copy. In addition, there may or may not be a newer version. Depending on those two variables, you pick:

• Good Copy, No new version: Nothing to do.
• Good Copy, New version: Patch update.
• Corrupted Copy, No new version: Repair.
• Corrupted Copy, New Version: Potential Full update.

Note: you can do this decision file per file.

Let us see how each one works:

---

Nothing to do.

Done.

---

Patch Update

You need to have precomputed, on the server, the patches from the old versions to the current one (e.g. with bsdiff, xdelta, etc...). Since this is on the server, both the last and the old version are available for comparison, so you can create the patch files. You would keep them around because a client may fall behind more than one update.

It works like this:

• Ask the server for the patch notes starting from the client version.
• The server looks for the patch files...
• If there are not patch files, return failure and fallback to Full update.
• If the patch files are available, return the list of patch files from the requested version to the current one.
• The client then downloads and applies the patches in order.
• Check version and integrity again at the end of the process.

---

Repair

To find the corruption, you can split the file in pieces of fixed size (e.g 4MB) and compute checksum corresponding to each piece. In the server, you would have detailed data about the files; this includes the length of the file and the checksums for the pieces.

This is the process:

• For the file that failed checksum, ask the server for detailed data.
• The server returns the size of the file and the checksums of the pieces.
• The client compares the size, if it does not match then truncate (or grow) the file to the right length.
• Compute the checksum of each piece of the file and compare it with the one you got from the server.
• If the checksum does not match, download that piece from the server and update it in the client.

---

Potential Full Update

You will run the full update (download the full file and replace it) for a file that is missing, or for a file for which there are no patch files. However, if you have a corrupted file, you need to decide if you want to repair it and the patch update or you want to download the full file and replace it. So, consider computing the piece checksums as part of the process that computes the full checksum of the initial check^... this way you will have them available already for the repair process.

^: For instance, the global checksum of the file can be the hash of the checksum of the pieces. Do not use partial checksums, doing that will make the checksums not match after the first corruption is found, and we don't want that.

This optimization will allow you to see how corrupted is the file before deciding if you want to repair it.

Now your criteria need to be based on:

• How corrupted is the file
• How much have the file changed (how big are the patches)

Compute how much do you have to download to fix the damaged pieces plus the patches, it is accounts for more than the full size of the file... download the full file instead. Otherwise, do a repair and then a patch update.

---

Addendum

You may want to add rules to avoid repairing and patching certain files. For example, do not bother repairing or patching files that are too small. In addition, you want to black list executable files, because many antivirus will not be happy with your updater editing executables (besides, executables should be small; the bulk should be in the assets).