3
\$\begingroup\$

I am currently researching resource streaming for my current game engine. To add a bit of context, I am doing a contiguous world for a Baulder's Gate style CRPG. So while the camera is usually looking downwards at an angle... it can be rotated and lifted up and down to compensate for mountains.

That means for vistas, it needs to be reasonably decent. At least with an added advantage that there will probably be very few unique assets that will be loaded in every terrain cell. (currently each cell is about 60m).

Most of the research I found turns up to have a separate thread for loading in assets. I can see how that works, as obviously holding such a thing on the main thread could introduce stalls.

But... currently my architecture is the main thread which has rendering and gamelogic. Then the rest are just worker threads for a job system.

I planned on moving the Rendering system off of the main thread, which means that on a common consumer PC (4 threads) I have two fixed threads... and two worker threads.

Moving a loader to it's own thread would basically reduce performance on some update cycles to a single threaded update.

My main thread is controlled by lua, and low level systems are in C++. Lua dispatches low level jobs only (Physics first, Animation updates, Transform updates, dispatch to renderer), and waits for them to finish before continuing to the next job dispatch, as certain data requires information from the previous component updates.

Annd... one recommendation I ran across, (because I am only targeting PC for now) was to keep data about each cell on the memory. Such as transforms, skeleton handles, material handles, etc.

So... I have a few questions. How well would a resource streamer work if each loading call was dispatch to a job as it is needed? From my research, I found multiple people stating that multiple async loading could slow the disk drive down.

What would be a few effective strategies for resource streaming? I know that it's better to have everything in one file, like a zip instead of scattered about folders. But I could never find any straight answers. One google search said to cut the file up into equally sized bits (which sounds logically stupid in my opinion.) another was priority loading, which sounds pretty reasonable. I understand this question is pretty vague, but finding answers in general in a world dominated by pre-built engines is already difficult enough.

\$\endgroup\$
  • 2
    \$\begingroup\$ Looks like you're confusion "cores" and "threads". A typical consumer PC has 4 cores, but runs hundreds of threads using pre-emptive multi-tasking. \$\endgroup\$ – Paul Houx Oct 10 '15 at 16:33
  • \$\begingroup\$ Ah thanks. Actually, I wish that answer came sooner. Because I spent a good long time trying to figure that out... before eventually learning that threads are actually scheduled by the OS for processing time. \$\endgroup\$ – moonshineTheleocat Oct 11 '15 at 1:19
4
\$\begingroup\$

I can give you only general information, based on my own experience. I have worked on AAA game engine development, but as a graphics programmer I was never involved with loading assets efficiently. We already had a system for that.

Yes, streaming assets in from disk should happen on a separate thread (or a few) and never on the same thread as your game logic or rendering. Disk operations are very slow compared to crunching data and loading things from memory.

Having more than one thread for asset loading will speeds things up (especially if you're loading assets from an SSD), but having too many of them may cause the drive to jump from sector to sector, slowing things down. Test, test, test on a number of different setups to determine the right number of loader threads.

Writing your own threaded asset loader does not have to be hard. Create a pool of a few threads, each running infinitely (until application quits). Use a concurrent container for communication between the main thread and the worker threads: your main thread pushes a file path onto the container and notifies the workers using a condition variable. One of the workers pops the file path and loads the asset. If possible (shared contexts) it uploads the asset to the GPU as well, but be careful not to stall the GPU. It then notifies the main thread that the asset is available and sleeps until it needs to load the next asset.

See for example: https://juanchopanzacpp.wordpress.com/2013/02/26/concurrent-queue-c11/

To speed up loading, use binary file formats (so you don't have to parse text files like OBJ or JSON). Compress your data. Load a few big chunks of data instead of lots of small chunks.

EDIT: should you use ZIP files for your data? It depends: loading compressed data is usually faster, provided that decompression is fast. If you have a C++ library that can do the decompression, using ZIP files is a good start. Note that it will not protect your data from prying eyes, as everybody can decompress ZIPs. If possible, come up with your own data file format.

A good source of information is the Urho3D engine. See e.g. their PackageFile class.

\$\endgroup\$
  • \$\begingroup\$ Freaking brilliant! I actually hadn't thought about that. And given that I need to load several chunks at once, it does make sense to use a few threads to load in data. \$\endgroup\$ – moonshineTheleocat Oct 11 '15 at 1:20
  • \$\begingroup\$ One final comment. Is it effective to use a zip. Or should I build my own .dat file... and do you have resources on the approach? \$\endgroup\$ – moonshineTheleocat Oct 11 '15 at 2:32
  • \$\begingroup\$ I have edited my answer with additional information. \$\endgroup\$ – Paul Houx Oct 11 '15 at 20:29
  • \$\begingroup\$ So from the looks of things, I think the Package File for Urho3D is all one directory. But the File names have been mangled by a hashing algorithm. So when you do a look up, you can use the file path of the file, but you are just comparing a 32bit value? Also, I've never built a package file of my own, so the concepts are fairly new \$\endgroup\$ – moonshineTheleocat Oct 11 '15 at 21:37
  • \$\begingroup\$ The PackageFile class reads a single, binary, file containing many separate assets. It's the result from compressing and writing data using the PackageTool. Maybe I should have pointed you to the tool instead of the reader, my apologies. I am not a Urho3D user myself, but do think it's a good source to learn more about this subject. \$\endgroup\$ – Paul Houx Oct 12 '15 at 3:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.