The fastest way to do this is store everything in one file and jump the cursor to the chunk you want to read. Once you hit the disk reading a sequence from that his point is pretty fast.
The multiple hits to different INodes to find the location of the file on the physical volume its whats taking most of the time and also what scales poorly.
Also, since this is dynamic, you're also going to need a map that stores the offset in the file for each chunk.
On Disk
[Chunk 1][Chunk 2][Chunk 3][Chunk 4][Chunk 5][Chunk 6][Chunk 7][Chunk 8][Chunk 9]
Visible
[7][8][9]
[6][1][2]
[5][4][3]
Then you just need to open a stream that reads from the file but doesn't lock other streams/processes from accessing it. Then you need to read from the correct offset for the correct distance. I believe in C# it's the below.
var chunk = new byte[4194304];
using (var file = new FileStream (openFileDialog1.FileName, FileMode.Open, FileAccess.Read, FileShare.ReadWrite))
using (var reader = new StreamReader (file, Encoding.Unicode)) {
reader.Read(chunk, offset * 4194304, 4194304);
}
}
Now because of the fact that you opened the stream in readonly mode and allow others to Read/Write to it you can keep adding new chunks to the end. Just keep track of their offset number and don't try to read them before they are there.
P.S.
you are not going to want to use the using block since you are going to want only 1 reading stream through the life of what ever level you are using. Also you'll probably have to save the chunk mapping to another file on exit but that is just one load when you load your level.
