So I'm creating a game and I would like to know how to perform resource streaming (rather than loading the whole file into the buffer) for things such as my maps, sounds, music, etc. I'm using C++ and DirectX9 if in case you need to know.
A note, setting off multiple reads will be slower because the disk I/O keeps seeking around to pull bits of data from each open request.
To get around this I created an intermediate streaming traffic cop that prioritized multiple async read requests and serialized them to the OS. In your own game you can create priorities based on data type or how soon the new data will be needed so that audio can come in first to avoid skipping and far away data will be loaded last.
A note on audio: you do want to stream audio because data is pretty large, but you want your game to stream the data before it's needed because waiting for the actual "Play sound" call before loading will cause all sorts of delay and spoil the user experience.
Since background loading and preparing data is a complex sequence of events involving threads, callbacks and voodoo that spans several systems in your game I suggest designing the interactions carefully before laying down any code to avoid massive debugging headaches later.
It's more than just streaming the assets — the disk itself is a mechanical device and you need to minimize seeks and simultaneous parallel streams. So you need to chunk up your data so it can be streamed en masse, and then evicted all at once without any risk of leaks.
Specifically re streaming world data, I believe it was GPG's Dungeon Siege that was the first well-known title to achieve this really efficiently for a really large 3D world. It enabled the modern MMO (post the whole tile-based Ultima Online era).
Here's an article where Scott Bilas outlines the solutions they used. I don't doubt it would help to do further searches on the same topic. I seem to recall a GamaSutra article, but not too sure. Could be the same one.
I am sure things have come a long way since then, but that should get you started.
You want to use the concept of chunks or pack files or the like where the entire chunk is streamed. The chunk should contain the data that it needs when used in the game. World Geo, specific models for that area, textures, sound effects, scripts and the like. It should not contain background audio, that should be a streaming system on its own.
You really dont want to read an individual packed file for each of these, but you definitely do not want to read each individual asset. The FileIO will kill any overhead. (Ideally the chunk files are all packed together into one large file you can just fseek() through to get to the right chunk you want to load)
Hope this helps.