Reading this article here, the author writes about chunk saving and using a lookup table and then saving each chunk in a region file, each region file having 4096 chunks and each chunk having 4096 bytes of data. He structures it so that each chunk in the region file is offset by 4096 bytes - perfect. So far so good, I've created this in C# easily.
My problem is now with compression. If I compress each chunk down, that is great, nonetheless I still have a region file that is full length in size based on each chunk being 4096 offset - so the file itself still looks full size despite each chunk being compressed and save within it's 4096 byte allotment. I don't want to compress the entire region file because that is extremely inefficient to have to always decompress 4096 chunks in a region just to get 1 chunk. It is better to compress each chunk individually before adding to region file.
Can someone suggest how I might work around this without resizing the region file? I do not want to resize a region file every time it is modified, which is also what the author of this article avoids too. File resizing would be too costly in performance.