1
\$\begingroup\$

So, I often see assets being packaged in some sort of custom container format.

World of Warcraft uses a bunch of files called "data.xxx" where xxx is a number (but they used something like MPQ files in the past. Not sure about the extension).

Warhammer Online used MYP files.

GW2 just had a giant .dat file (I think) with literally everything in it. Like, the whole game is one executable and then a giant file and that's it.

But is there a benefit apart from encryption? The only thing I can think of is that I can give assets something like an identifier which the engine then looks up in some sort of index that contains size and offset and then it reads from a giant file.

A few examples: I load texture "skybox-dessert-world" and it loads the texture. I don't have to worry about the format or the path. I just convert to the format I want in my packager.

Maybe something that makes more sense: I reference textures and height maps in my terrain file format. Using identifiers, I can reduce the length of the file name. Especially if I'd have to use the same texture file multiple times.

Or maybe something like this: I have multiple version of a model for discrete level of detail and just loading "model-npc-elf-male" would load the appropriate models.

But is that worth it? Or am I making a big deal out of nothing and loading files straight from disk is just as good? I've heard it's a good thing in the context of encryption and compression but that's about it.

\$\endgroup\$

1 Answer 1

2
\$\begingroup\$

Only you can answer if it is worth it for your own projects. It is more work to build a container file format yourself, although you could just use .zip (maybe renaming the extension) and one of the many available APIs to access zip-compressed files.

Some reasons you might want to create an archive or container file are:

  • obfuscation of your data (as you already pointed out) to help defeat, somewhat, casual hacking or inspection
  • size optimization; one (or few) big files can take up less space than the same data split across many small files because there is a minimize file size representable by the filesystem (which includes header and metadata information about the files, which scales up in total with the number of distinct files)
  • performance optimization; in a custom archive format you can lay out the subfiles however you like, grouping related files near each other so as to minimize disk seeks during IO operations
  • simplicity; it can be easier to install and manage one file versus many

For all of these potential benefits, you pay a cost in complexity (which really means that that "simplicity" point is about moving the simplicity around, not necessarily increasing net simplicity) to design or at least implement such a format. So you need to weigh the value of the specific advantages you want to leverage against the cost of building and maintaining the feature to provide them for your own projects.

\$\endgroup\$
2
  • \$\begingroup\$ Didn't even think about the headers of the file system and minimising seeks. Thanks \$\endgroup\$
    – Asyx
    Aug 17, 2016 at 10:55
  • 1
    \$\begingroup\$ That, and to open a file the OS needs to do additional checks such as security. Having an open archive file doesn't need that; you can just seek through the file. \$\endgroup\$
    – Caramiriel
    Aug 23, 2016 at 14:36

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .