Why is custom memory management needed for assets?

Question

Pretty much all resources relating to game programming, especially 3D open-world games, talk about how you have to constantly be unloading and reloading assets to and from disk, system memory, and video memory. I can understand this on consoles, as they have very simple memory management schemes that cannot deal with any potential overflow.

However, on the PC, the situation is very different. There is virtual memory provided by the OS for system memory and the graphics driver handles swapping back and forth from the CPU to the GPU. So why is it that it's not sufficient to load everything at once and only have to deal with prefetching and possibly making sure that there is no blocking while a required asset is being swapped?

---

This is not to say that everything is being allocated naively using something like malloc. Everything will be kept contiguous and cache-coherent. It's arguably easier to make your memory access efficient if you don't have to worry about paging in and out manually anyways...

---

UPDATE:

After reading a bit into asynchronous buffer transfers, I get the idea that the way the graphics driver can automatically page memory in and out is suboptimal. Does this also hold true for CPU resources; i.e., is it better to always manually manage when a given resource is to be loaded and unloaded even in the presence of virtual memory?

Answer 1

Modern open-world games simply don't fit in memory. Keep in mind that most games are still 32-bit due to the number of gamers with 32-bit OSes, and a 32-bit process at best can only have 4GB of addressed memory at a time (independent of virtual memory) but realistically is limited to 2-3 GB. Toss in fragmentation and the actual amount of usable objects you can have in a memory at a time is a fair bit smaller than the total data you need for a large environment. Even on a 64-bit OS, you often need to target low-spec machines with only 4GB of memory to ensure maximum end-user compatibility. Let's not forget mobile devices, "under-powered" hardware like the WiiU, etc. Developers have to target a lot more than big fancy gaming PC rigs.

Virtual memory is not a solution, firstly because it doesn't negate address space limitations and also because it is not optimal. The OS can page things out at inopportune times or in inopportune ways. This was a problem dealt with by OSes when hibernation first came around; paging out the processes to disk and then paging them back in was significantly slower in some cases than having the application explicitly stream the resource back in. You can still see the problems today when a system is running out of memory, pages are paged out to disk, and the system becomes unresponsive becomes critical things necessary to run your WM/explorer.exe get paged out. Virtual memory was a solution devised to a problem decades ago where systems had very little RAM and the gap between memory and disk speeds was much smaller. Even today's fastest SSDs are a tiny fraction of the speed of cheap/slow RAM, and trying to pretend that you can use your disk as virtual RAM just isn't the best idea anymore. This is an age where some people do the opposite and load up file systems into RAM disks; the constraints that led to the creation of virtual memory don't hold true anymore.

Even aside from disk issue, custom resource management is still a big part of game development. From debugging hooks to fragmentation concerns to intelligent streaming of complex resources, simply pretending the whole world is an infinitely big box of bits is not realistic.

Answer 2

For a 32-bit game, as most games are for a variety of reasons, even a game that comes on one single-sided DVD (4.3GB) already has far more content that can be fitted into a 32-bit address space. And that's assuming the content isn't compressed on disc, and a perfectly optimal, load everything at once into contiguous address space approach. Many games now come on multiple DVDs, easily surpassing 10GB of content. In my experience on PC, once your address space approaches 2GB allocations will start to fail, and that becomes a hard limit, regardless of how much physical memory the user has.

If you do make the jump to 64-bit, assuming your engine supports it, that hard limit goes away, only to be replaced by an effective soft limit - as soon as virtual memory swapping begins to happen, the performance of the game will drop unacceptably. If the main process ever has to wait for memory to be paged back in, the frame-rate will stutter and glitch. Virtual memory might make that transparent, but it doesn't make it performant. It physically takes time to get that memory back from disk, and while you are waiting, your game has stalled. While it's possible to work around (with background pre-fetching of memory pages that you know you'll need soon), it's a difficult and unreliable process, and will still have performance implications.

The OS's algorithms for deciding what pages to swap out to disk have no knowledge of your game and what it is likely to need next. Furthermore, the OS still has to support other applications running simultaneously with your game, as well as the OS itself. Having to stall while pages are returned from disk is a massive performance hit, and it doesn't matter whether it's your game, another application or the OS that incurs it, the game's performance will suffer massively.

So regardless of whether the memory limit is soft or hard, memory is still limited. But even if you could just load everything at once, without hitting a limit of performance, there are still reasons why you wouldn't want to:

1. Your game takes far longer to load and get to an interactive state than it needs to, because it simply loads everything at the start and keeps it resident. You can load about 150MB/sec from a hard disk, and even the fastest DVD drive (24x) loads at 33MB/sec. That means the example 4.3GB DVD worth of content will take at least 30 seconds to load from HDD, and over 133 seconds to load from DVD. Those are unacceptably long load times.

2. Your game takes much more memory than is justified. If only 10% of the content is visible at any one time, holding the rest resident is simply wasteful, and prevents the user from using that memory for something else. Users are generally unforgiving of games which require a disproportionately large amount of memory to run properly.

Streaming assets from disk dynamically, whether in the background or synchronously during a loading screen, allows you to be far more efficient in your use of memory, with negligible effort and only a minor downside for the user. But it's an optimisation, and like any other optimisation it has diminishing returns the more you try to apply it. At some point the increase in memory footprint for keeping something resident is outweighed by the advantages gained from having it there without the need to load. But until that point is reached, it's better to load only what you need, when you need it.

Answer 3

Just speaking very generally and simplistically, and without considering the details of virtual memory implementations, the developer always has knowledge in foresight that the VM implementation lacks.

The developer can always say, "I don't need to load this audio file right now. The music inside is only used for the game over screen." And immediately after the game over screen, the developer can say, "I don't need this audio clip anymore in physical memory. It's only used for this game over screen."

The OS has no such foresight. It might be able to figure out, many page faults later, that some audio clip is no longer needed in physical memory since it hasn't been accessed in a good while. But turning foresight into hindsight translates into lots of page faults, and lots of page faults translates to hiccups in frame rates in a software as time-critical as a video game. There the foresight of the developer really helps if you want to avoid such hiccups.

And that applies conceptually regardless of the hardware and software. Assuming paging in memory is expensive, then the developer's foresight will always help in reducing that expense.

Speaking even more broadly, there's a never-ending cycle between hardware designer, compiler designer, OS/driver designer, and application developer. The hardware/compiler/OS/driver developers often try to implement optimizations to speed up your average application based on its usual memory access patterns, and perhaps sometimes with the hope like, "People should just be able to write code however they want and it should be fast." But if there was any thought of that type going on, it usually fails for performance-critical fields, because then the performance-critical developers start learning the intricate details of their compiler, hardware, OS, drivers, etc. and start writing code specifically designed to exploit that as much as possible to write the fastest code possible (like prefetches, hot/cold field splitting, SoAs, etc. for cache-friendly code). And that's like a game that never ends. These things are never treated as black boxes in performance-critical fields, since the developers are competing for performance.

Personally I kinda wish virtual memory didn't exist since it adds an additional layer of performance unpredictability in ways that tend to be too extreme and incur way, way too much of a performance penalty when things go really south to the point of unusability. I have sometimes run into cases where I used some application where I typed an extra digit or two by accident when drunk into some input field which then caused it to exhaust physical memory so quickly in ways that brought the OS to a crawl where I couldn't even click the cancel button on the progress bar anymore and had to wait 10 minutes while jamming Ctrl+Alt+Del to kill a process and cursing to myself while spilling my drink to get back to a point where things are usable again, and that was in spite of the page file being stored on an SSD. In those cases I would have rather preferred an "out of memory" error or something, at which point I might close some of my 17 tabs of porno (it's okay, I bookmark my favorites anyway) to free some memory and then immediately go about resuming my business.