Reading this article here, the author writes about chunk saving and using a lookup table and then saving each chunk in a region file, each region file having 4096 chunks and each chunk having 4096 bytes of data. He structures it so that each chunk in the region file is offset by 4096 bytes - perfect. So far so good, I've created this in C# easily.

My problem is now with compression. If I compress each chunk down, that is great, nonetheless I still have a region file that is full length in size based on each chunk being 4096 offset - so the file itself still looks full size despite each chunk being compressed and save within it's 4096 byte allotment. I don't want to compress the entire region file because that is extremely inefficient to have to always decompress 4096 chunks in a region just to get 1 chunk. It is better to compress each chunk individually before adding to region file.

Can someone suggest how I might work around this without resizing the region file? I do not want to resize a region file every time it is modified, which is also what the author of this article avoids too. File resizing would be too costly in performance.


Why don't you create a smaller allocation unit size (instead of 4096 per chunk)?

From what I understand, each chunk is 4096 byte long uncompressed. I'm guessing that a lot of data in the region is the same, so compressed it's at least 20x smaller, so that's about 200 something bytes. You can compress a bunch of chunks and chose allocation unit size that would be small enough, but still fit most of the compressed chunks.

If you make a allocation unit size 256 bytes you could store most of compressed chunks into a single unit, filling it with data and then completing the unit by adding zeroes up to 256 bytes. If the chunk happens to be larger than that, you'd have to use two or more units to store it.

In that case, your offsets would get messed up and you would have to create a header that would store the offset of each chunk from the start of the file in regards of number of 256 byte blocks. Given that you have 4096 chunks, you could get away with 2 bytes per chunks.

Alternative is to choose a unit size that you'd know for certain wouldn't get too small for a compressed chunk, but that's a gamble.

I know this is a late answer, but it might help someone else who stumbles upon it.


You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .