# How can I store all my level data in a single file instead of spread out over many files?

I am currently generating my level data, and saving to disk to ensure that any modifications done to the level are saved.

I am storing "chunks" of 2048x2048 pixels into a file. Whenever the player moves over a section that doesn't have a file associated with the position, a new file is created.

This works great, and is very fast. My issue, is that as you are playing the file count gets larger and larger.

I'm wondering what are techniques that can be used to alleviate the file count, without taking a performance hit. I am interested in how you would store/seek/update this data in a single file instead of multiple files efficiently.

• You'd basically have to write a miniature file system to be able to store everything in a single file. It'll add to the complexity and might not be worth it. – thedaian Jan 20 '12 at 19:24
• I don't really have much experience with the following, but perhaps a file-based nosql db (stackoverflow.com/questions/2403174/…) could be an option. – Chris Jan 20 '12 at 19:32
• How many files are we talking about here? Are players generating tens of thousands of files, or just hundreds? What are players changing in the chunk? Is chunk generate an expensive step (ie, do you need to cache the whole chunk vs just a diff?) – Leniency Jan 20 '12 at 20:55
• Minecraft went through this conversion at some point too. I believe it started as a mod then was incorporated into the main build. Worth looking into. minecraftwiki.net/wiki/Region_file_format – MichaelHouse Jan 20 '12 at 23:07
• @thedaian a little more complexity but you can do some cool stuff and really cut down on seek times if your willing to some extra work in memory so the filesystem doesn't have to do it on the disk. – ClassicThunder Jan 21 '12 at 5:29

The fastest way to do this is store everything in one file and jump the cursor to the chunk you want to read. Once you hit the disk reading a sequence from that his point is pretty fast.

The multiple hits to different INodes to find the location of the file on the physical volume its whats taking most of the time and also what scales poorly.

Also, since this is dynamic, you're also going to need a map that stores the offset in the file for each chunk.

On Disk

[Chunk 1][Chunk 2][Chunk 3][Chunk 4][Chunk 5][Chunk 6][Chunk 7][Chunk 8][Chunk 9]


Visible

[7][8][9]
[6][1][2]
[5][4][3]


Then you just need to open a stream that reads from the file but doesn't lock other streams/processes from accessing it. Then you need to read from the correct offset for the correct distance. I believe in C# it's the below.

var chunk = new byte[4194304];
}
}


Now because of the fact that you opened the stream in readonly mode and allow others to Read/Write to it you can keep adding new chunks to the end. Just keep track of their offset number and don't try to read them before they are there.

P.S. you are not going to want to use the using block since you are going to want only 1 reading stream through the life of what ever level you are using. Also you'll probably have to save the chunk mapping to another file on exit but that is just one load when you load your level.

• This is giving me some great ideas. Would this method require you to write each chunk the exact same number of bytes? I would imagine, as the offset seeking would require it. – jgallant Jan 21 '12 at 13:35
• As long as you have the first byte and how long each chunk is they don't have to be equal size. You just have to have something in memory keep track of these two pieces of data. – ClassicThunder Jan 21 '12 at 15:46
• Of course, if you didn't have them of equal byte size, you'd have to do a lot of shifting if one was to grow in size. – MichaelHouse Jan 23 '12 at 23:09

Depending on the time required to generate a chunk, you could just store diffs or a current state (enemy locations, etc). As the player moves back into a chunk, it generates again using a stored seed, then loads any changes that were made from the file.

If players are allowed to make significant changes, this might be slow and the diff file will still be fairly large, but for only small changes, should be an inexpensive operation. Multiple chunk diffs could be consolidated into a single file as well - something sized reasonably that could be loaded into memory.

You probably wouldn't want to move all the diffs into a single file though - that opens up a number of other issues with memory, or changing the middle of the file.

I know this is a pretty old thread - but I'd just like to chime in that I think a ZIP archive might be the best way to go here. You get compression with your data (if you're using raw bitmaps, especially), readability in the operating system, and you get the single file as you wanted.

What about a dir scan to check the time-stamp of the files in the level data directory vs the current active file and giving a grace of the previous file and the file going forward every 10 seconds or so, and whatever isn't being used just delete them.

Unless you require the player to go back. Then just cleanup the level data upon level completion or checkpoint? Could get large for sure, but I don't think there are many options available here

How about multiple chunks per file? You say your chunks are 2048 x 2048, how about putting 16384 x 16384 in a file. Flag which ones exist somehow so you know whether you need to create it.

If you can generate the chunks fast enough as the player explores anyway, then you don't need to cache them on disk at all; all you need to do is store the seed for the perlin noise functions you are using to generate your procedural content again on demand.

These can be stored in a single file, and can be written sequentially and sorted in RAM when loaded; there's no need for a complicated sorted structure in the file itself on disk. You can be reading from it only on startup, and writing to it as you generate new 'pages' (as they are termed) in the game-world.