First of all, are you sure you really need that? Have you calculated the memory footprint?
A small back-of-the-envelope calculation: A single mob and its state should fit into 100 byte of data. Let's give it a whole kByte, in case you are doing something extraordinary. When a cell has 1000 such entities, it requires a MByte. If your world is 100x100 cells, ...
One way you could solve this problem is not actually storing state on disk, but just setting up your generation code to use a seed for the random number generator, so it generates the same thing for a given area every time the area is generated, deterministically.
Then you just keep the 1000 or so most recently visited areas in memory. When areas are ...
Creating a seamless open world without loading bars is not an easy task. There are lots of small and large problems which need to be solved. This is not a beginners project!
The basic approach is usually to divide the world into sections and only load those sections which are around the player. When the player moves around the world, new sections and their ...
Save it to the hard drive instead.
Saving large quantities of data in a non-volatile fashion is literally why hard drives were invented. If a player isn't interacting with it anymore, you won't need to rapidly access it, so why keep it in the RAM? Just save it to the your server's hard drive in a text file or something.
Instantiate seems to be the culprit here. You're trying to instantiate THOUSANDS of new game objects in a single frame. I would recommend to delay the instantiate by batches of 100 per frame.
And I'm not even mentionning the thousands of new pos4d and new Vector3 that are called.
To address your initial concern, unless you are targeting older generation consoles or embedded devices, you probably don't need to worry about fragmentation.
With that said, one way you can address the problem is to have a dedicated memory pool for strings which are then interned.
With string interning in C, you create a hash table where the keys are ids ...
Pombal gives a very good answer, but I'd just like to add a note to it.
Like your textures, sounds, and other game-related data, strings are also a resource. Lots of game engines allow you to load and unload resources in groups... that is, you can have a global group (this stuff is alive for the duration of a play session), and then more specific groups ...
Your crashes is mostly because of improper handling of the object's ownership.
When you add something to a collection, you should clearly understand who's owning the object. Is that handler owns the added objects now, and may delete those when time has come? Or is that handler only references the objects, and there's some other subsystem that will destroy ...
For a project that we worked on, we had utilized grouping. It was one creature (with a model of many creatures), but had many hitboxes. When a hitbox HP became 0, it would divide the creature and create a second or third model.
Utilize low-poly models. We had to use this a long time ago for massive battles with multiple mobs.