1
\$\begingroup\$

So, I often see assets being packaged in some sort of custom container format.

World of Warcraft uses a bunch of files called "data.xxx" where xxx is a number (but they used something like MPQ files in the past. Not sure about the extension).

Warhammer Online used MYP files.

GW2 just had a giant .dat file (I think) with literally everything in it. Like, the whole game is one executable and then a giant file and that's it.

But is there a benefit apart from encryption? The only thing I can think of is that I can give assets something like an identifier which the engine then looks up in some sort of index that contains size and offset and then it reads from a giant file.

A few examples: I load texture "skybox-dessert-world" and it loads the texture. I don't have to worry about the format or the path. I just convert to the format I want in my packager.

Maybe something that makes more sense: I reference textures and height maps in my terrain file format. Using identifiers, I can reduce the length of the file name. Especially if I'd have to use the same texture file multiple times.

Or maybe something like this: I have multiple version of a model for discrete level of detail and just loading "model-npc-elf-male" would load the appropriate models.

But is that worth it? Or am I making a big deal out of nothing and loading files straight from disk is just as good? I've heard it's a good thing in the context of encryption and compression but that's about it.

\$\endgroup\$
2
\$\begingroup\$

Only you can answer if it is worth it for your own projects. It is more work to build a container file format yourself, although you could just use .zip (maybe renaming the extension) and one of the many available APIs to access zip-compressed files.

Some reasons you might want to create an archive or container file are:

  • obfuscation of your data (as you already pointed out) to help defeat, somewhat, casual hacking or inspection
  • size optimization; one (or few) big files can take up less space than the same data split across many small files because there is a minimize file size representable by the filesystem (which includes header and metadata information about the files, which scales up in total with the number of distinct files)
  • performance optimization; in a custom archive format you can lay out the subfiles however you like, grouping related files near each other so as to minimize disk seeks during IO operations
  • simplicity; it can be easier to install and manage one file versus many

For all of these potential benefits, you pay a cost in complexity (which really means that that "simplicity" point is about moving the simplicity around, not necessarily increasing net simplicity) to design or at least implement such a format. So you need to weigh the value of the specific advantages you want to leverage against the cost of building and maintaining the feature to provide them for your own projects.

\$\endgroup\$
  • \$\begingroup\$ Didn't even think about the headers of the file system and minimising seeks. Thanks \$\endgroup\$ – Asyx Aug 17 '16 at 10:55
  • 1
    \$\begingroup\$ That, and to open a file the OS needs to do additional checks such as security. Having an open archive file doesn't need that; you can just seek through the file. \$\endgroup\$ – Caramiriel Aug 23 '16 at 14:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.