Managing large amounts of data and processes for an economy management game

Question

I'm making a management game in the vein of Anno or Stellaris, but I'm unsure how to manage all my data efficiently.

Short example: the player controls multiple islands. Each island has factories on it. A factory has inputs, a process that does something with the inputs, and outputs. The process depends on the inputs and the outputs and other parameters. The outputs could be another factories input. A factory needs human workers. Human workers have stamina ... ... ...

So the question is: how do you manage so much data and processes programmatically? And it's not only about running many processes simultaneously, they also depend on each other and the data, use each other, inform each other - on each tick. You also need the ability to get information out of those sub processes and put data into it at any time.

I'm struggling to come up with a system/codebase that could handle those things. I've choose C#/dotnet core since I assume it has something to do with threading, but I still haven't got anything that works.

Answer 1

I can't say anything about these particular two games, because they aren't open source and I wasn't involved in their development.

But usually when you have game mechanics with a large number of entities each doing relatively simple computations, then you usually use an Entity - Component - System architecture.

The basic gist is this:

• You divide the data required for the functionality of each entity in your game into granular components.
• You create an array for each type of component to ensure that they are in a connected memory section. This improves CPU performance when performing an operation on all components of the same type.
• You implement each game mechanic in form of a system which operates on one or more component-arrays.

Each island has factories on it. A factory has inputs, a process that does something with the inputs, and outputs. The process depends on the inputs and the outputs and other parameters. The outputs could be another factories input. A factory needs human workers. Human workers have stamina

If you would implement this in an ECS architecture, then you would usually implement it something like this (details can vary depending on the complexity of these mechanics):

• The WorkerStaminaSystem updates the stamina of all workers
• The WorkerProductivitySystem updates the productivity value of all workers depending on their current stamina and other values.
• The FactoryProductivitySystem updates the productivity of all factories depending on their current worker productivity and other attributes of the factory
• The FactoryProcessSystem updates resource stockpiles based on the productivity of all factories

The goal is to have not one large update loop which iterates each entity and does everything that needs to be updated. It instead has many separate update-loops which perform a pretty small operation on a large set of pretty small objects. Why?

• It is much easier for the optimizers of your compiler and the internal optimization of your CPU to process short loops than long loops.
• It is easier on the CPU caches if the data processed by a loop is as small as possible and in a consecutive memory section.
• It is possible to parallelize these update-loops over multiple CPU cores. For example, when your WorkerStaminaSystem has 100,000 workers and you have 4 CPU cores, then each core can process 25,000 of them. You won't need to worry about thread safety or implement expensive thread synchronization techniques, because each worker can be updated without knowing anything about the other workers and you only need read-access at most for the rest of the world.
• It is often possible to run multiple systems in parallel. The limited amount of components processed by each system makes it easy to recognize systems which operate on completely independent data-sets. Those can be run in parallel without having to worry about thread safety.
• It allows to update systems in different intervals. Some systems might not need to run every single update-tick. Perhaps it is sufficient to update the worker stamina just every second or even less? You can even have systems which perform staggered updates. Only update a limited chunk of objects each tick to avoid lag spikes.