0
\$\begingroup\$

With a custom binary file format for archives that is structured similiar to Valve's VPK files I have a Game.index.gcf which holds information about all files and where exactly they are stored. I can split up and say, that the /Assets/Maps and /Assets/Scripts folders should be stored in Game.000.gcf while the bigger /Assets/Music folder could be split up into Game.001.gcf and Game.002.gcf. This should boost performance but only if done right. Moreover with updates I could say, just update this particular split archive and not the entire archive (when storing everything in one really huge a file).

As of yet, I don't know how to write such an adaptive algorithm which automatically decides the best way to split all archives and files. What aspects should I keep in mind and what should I be aware of when creating these? Do you have any suggestions?

\$\endgroup\$
3
  • \$\begingroup\$ Why do you think this would boost performance? If you mean that there is no need to read the whole file, then proper use of seeks within a large file would have the same effect. \$\endgroup\$
    – Liosan
    Commented Mar 28, 2013 at 8:15
  • \$\begingroup\$ Besides, an "adaptive algorithm" means that archive layout would change often whenever you add new content, which doesn't sound fun even for the dev version. \$\endgroup\$
    – Liosan
    Commented Mar 28, 2013 at 8:16
  • \$\begingroup\$ @Liosan: I was thinking that seeking to and reading a small file in a 5MB archive is much more efficient than doing this in one of 1GB size which hold everything. (the sizes are chosen arbitrary) Am I wrong? \$\endgroup\$
    – Christian Ivicevic
    Commented Mar 28, 2013 at 11:47

1 Answer 1

Reset to default
0
\$\begingroup\$

I wouldn't expect a performance difference between reading a 4KB chunk from a small file or from a big file, assuming both are properly implemented.

For your 1GB "huge" file, you could hold an in-memory index containing available filepaths stored in the VFS (or hashes of them, to store space). Each index entry would just contain the offset in the file - a read is a easy seek, just read the part of the file that you need. If you have compression applied, make sure each file is compressed separately or in series of smaller files, so that you can decompress your chosen file with little effort.

From the operating system point of view, it doesn't really make any difference if the files are 5 or 500 MB. Small files, that is below 10KB or so, actually have a negative impact (because finding the file and opening it also takes time).

I don't see any immediate advantage of using small archives instead of one big one, performance wise. In fact, many AAA games use this method of huge monolithic files.

The "easy patching" argument still applies, but then I guess you don't need any fancy "packer" algorithm.

\$\endgroup\$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .