Resource streaming with job threading system

Question

I am currently researching resource streaming for my current game engine. To add a bit of context, I am doing a contiguous world for a Baulder's Gate style CRPG. So while the camera is usually looking downwards at an angle... it can be rotated and lifted up and down to compensate for mountains.

That means for vistas, it needs to be reasonably decent. At least with an added advantage that there will probably be very few unique assets that will be loaded in every terrain cell. (currently each cell is about 60m).

Most of the research I found turns up to have a separate thread for loading in assets. I can see how that works, as obviously holding such a thing on the main thread could introduce stalls.

But... currently my architecture is the main thread which has rendering and gamelogic. Then the rest are just worker threads for a job system.

I planned on moving the Rendering system off of the main thread, which means that on a common consumer PC (4 threads) I have two fixed threads... and two worker threads.

Moving a loader to it's own thread would basically reduce performance on some update cycles to a single threaded update.

My main thread is controlled by lua, and low level systems are in C++. Lua dispatches low level jobs only (Physics first, Animation updates, Transform updates, dispatch to renderer), and waits for them to finish before continuing to the next job dispatch, as certain data requires information from the previous component updates.

Annd... one recommendation I ran across, (because I am only targeting PC for now) was to keep data about each cell on the memory. Such as transforms, skeleton handles, material handles, etc.

So... I have a few questions. How well would a resource streamer work if each loading call was dispatch to a job as it is needed? From my research, I found multiple people stating that multiple async loading could slow the disk drive down.

What would be a few effective strategies for resource streaming? I know that it's better to have everything in one file, like a zip instead of scattered about folders. But I could never find any straight answers. One google search said to cut the file up into equally sized bits (which sounds logically stupid in my opinion.) another was priority loading, which sounds pretty reasonable. I understand this question is pretty vague, but finding answers in general in a world dominated by pre-built engines is already difficult enough.

Answer 1

I can give you only general information, based on my own experience. I have worked on AAA game engine development, but as a graphics programmer I was never involved with loading assets efficiently. We already had a system for that.

Yes, streaming assets in from disk should happen on a separate thread (or a few) and never on the same thread as your game logic or rendering. Disk operations are very slow compared to crunching data and loading things from memory.

Having more than one thread for asset loading will speeds things up (especially if you're loading assets from an SSD), but having too many of them may cause the drive to jump from sector to sector, slowing things down. Test, test, test on a number of different setups to determine the right number of loader threads.

Writing your own threaded asset loader does not have to be hard. Create a pool of a few threads, each running infinitely (until application quits). Use a concurrent container for communication between the main thread and the worker threads: your main thread pushes a file path onto the container and notifies the workers using a condition variable. One of the workers pops the file path and loads the asset. If possible (shared contexts) it uploads the asset to the GPU as well, but be careful not to stall the GPU. It then notifies the main thread that the asset is available and sleeps until it needs to load the next asset.

See for example: https://juanchopanzacpp.wordpress.com/2013/02/26/concurrent-queue-c11/

To speed up loading, use binary file formats (so you don't have to parse text files like OBJ or JSON). Compress your data. Load a few big chunks of data instead of lots of small chunks.

EDIT: should you use ZIP files for your data? It depends: loading compressed data is usually faster, provided that decompression is fast. If you have a C++ library that can do the decompression, using ZIP files is a good start. Note that it will not protect your data from prying eyes, as everybody can decompress ZIPs. If possible, come up with your own data file format.

A good source of information is the Urho3D engine. See e.g. their PackageFile class.