I am currently researching resource streaming for my current game engine. To add a bit of context, I am doing a contiguous world for a Baulder's Gate style CRPG. So while the camera is usually looking downwards at an angle... it can be rotated and lifted up and down to compensate for mountains.
That means for vistas, it needs to be reasonably decent. At least with an added advantage that there will probably be very few unique assets that will be loaded in every terrain cell. (currently each cell is about 60m).
Most of the research I found turns up to have a separate thread for loading in assets. I can see how that works, as obviously holding such a thing on the main thread could introduce stalls.
But... currently my architecture is the main thread which has rendering and gamelogic. Then the rest are just worker threads for a job system.
I planned on moving the Rendering system off of the main thread, which means that on a common consumer PC (4 threads) I have two fixed threads... and two worker threads.
Moving a loader to it's own thread would basically reduce performance on some update cycles to a single threaded update.
My main thread is controlled by lua, and low level systems are in C++. Lua dispatches low level jobs only (Physics first, Animation updates, Transform updates, dispatch to renderer), and waits for them to finish before continuing to the next job dispatch, as certain data requires information from the previous component updates.
Annd... one recommendation I ran across, (because I am only targeting PC for now) was to keep data about each cell on the memory. Such as transforms, skeleton handles, material handles, etc.
So... I have a few questions. How well would a resource streamer work if each loading call was dispatch to a job as it is needed? From my research, I found multiple people stating that multiple async loading could slow the disk drive down.
What would be a few effective strategies for resource streaming? I know that it's better to have everything in one file, like a zip instead of scattered about folders. But I could never find any straight answers. One google search said to cut the file up into equally sized bits (which sounds logically stupid in my opinion.) another was priority loading, which sounds pretty reasonable. I understand this question is pretty vague, but finding answers in general in a world dominated by pre-built engines is already difficult enough.