Population dynamics is a research field for over 200 years and describes changes in size and age of populations over time. Some famous models include Lotka-Volterra equations as a model for predator and prey.
Some assumptions in the Lotka-Volterra model are:
- The prey population finds ample food at all times.
- The food supply of the predator population depends entirely on the size of the prey population.
- The rate of change of population is proportional to its size.
- During the process, the environment does not change in favour of one species and the genetic adaptation is sufficiently slow.
- Predators have limitless appetite
They naturally lead to oscillations (with a fixed delay between them) of both involved species if the parameters are chosen right.
Otherwise you can also get one or both of them to extinction.
You can probably model this in a cellular automaton fashion (e.g. as agents) so that it directly fits into your game.
Some advice to get the system more stable:
- Have prey and predator species (so species that just reproduce and species that need to kill others in order to live and reproduce)
- Have a limited but renewable supply of food for the prey.
- Rate of change of population is proportional to its size, prey reproduces itself faster than predators (otherwise prey will die out)
- Keep that predators have limitless or virtually limitless apetite (otherwise overpopulation of prey may follow)
- Have some "refuge areas" for the prey where predators seldom go
- If prey dies out by chance (can happen in real life see Dodos) just put them back or cheat (make them reproduce faster if the population is very low)
- Have large enough numbers of prey to be statistically on the save side
To make it more interesting/diverse:
- Change the environment periodically and make the system (reproduction rate, agility, hunting success rate, ...) dependent on the environment
- The more stable you are the less the system will react on random influences/distortions
- Therefore keep the oscillations inbuilt in the system and the random fluctuations small enough, so you will always go back to equilibrium
- To have an additional security mechanism to ensure stability cheat (make species reproduce faster and more difficult to hunt if they are far below their equilibrium, make them reproduce slower, ... if they are far above)