Generally, you don't handle out-of-memory. The only sane option in software as large and complex as a game is to just crash/assert/terminate in your memory allocator as soon as possible (especially in debug builds). Out-of-memory conditions are tested for and handled in some core system software or server software in some cases but not usually elsewhere.
When you have an upper memory cap you instead just ensure that you never need more than that amount of memory. You can keep a maximum number of allowed NPCs at a time, for instance, and simply stop spawning new non-essential NPCs once that cap is hit. For essential NPCs you can either have them replace non-essential ones or have a separate pool/cap for essential NPCs that your designers know to design around (e.g. if you can only have 3 essential NPCsa, the designers will not put more than 3 in an area/chunk - good tools will help designers do this properly and testing is essential of course).
A really good streaming system is also important especially for sandbox games. You don't need to keep all NPCs and items in memory. As you move through chunks of the world new chunks will be streamed in and old chunks streamed out. These will generally include NPCs and items as well as terrain. Design and engineering caps on item limits need to be set with this system in mind, knowing that at most X old chunks will be kept around and pro-actively loaded Y new chunks will be loaded, so the game needs to have space to keep all the data of X+Y+1 chunks in memory.
Some games do attempt to handle out-of-memory situations with a two-pass approach. Keeping in mind that most games have a lot of technically unnecessary cached data (say, the old chunks mentioned above) and a memory allocation might do something like:
This is a last-stop measure to deal with unexpected situations in release but during debugging and testing you should probably just immediately crash. You don't want to have to rely on this kind of stuff (especially because dumping the caches may have some serious performance consequences).
You might also consider dumping high-res copies of some data, for instance you might dump the higher-resolution mipmap levels of textures if you are running low on GPU memory (or any memory in a shared-memory architecture). This usually requires a lot of architectural work to make worth it, though.
Note that some very unlimited sandbox games can be rather easily just crashed, even on PC (remember that the common 32-bit apps have a limit of 2-3GB of address space even if you have a PC with 128GB of RAM; a 64-bit OS and hardware allows more 32-bit apps to run simultaneously but can't do anything to make a 32-bit binary have a larger address space). In the end, you either have a very flexible game world that will need unbounded memory space to run in every case or you have a very limited and controlled world that always works perfectly in bounded memory (or something somewhere in between).