# How does Dwarf Fortress keep track of so many entities without losing performance?

In Dwarf Fortress you can have hundreds of Dwarves, animals, goblins, etc in game at any one time, each with their own complex AI and pathfinding routines. My question is how does this not produce noticeable slowdown? Does each Dwarf run in its own thread?

• Interesting question. Though I don't like the way it's phrased, since it would just be speculation to answer this question as worded. Anyway, you may have seen this article talking with Tarn Adams. Where they briefly cover path finding.
– House
Jul 23 '12 at 13:45
• It absolutely produces noticeable slowdown once you have a complex tunnel topology and 100+ dwarves. Jul 23 '12 at 17:48
• Yes, DF in my experience is incredibly slow in the scenario you describe. Jul 23 '12 at 17:48
• Destroy a primary stairwell and watch your framerate drop like a rock for a moment while every dwarf suddenly repaths at the same time. Jul 23 '12 at 20:13
• I'm going to chime in and agree that DF is not the best example; in fact DF is well-known to suffer performance issues. Feb 25 '13 at 21:13

Any system which had a thread for each of so many characters would run out of resources very quickly. Threads may give you access to extra processor cores but they don't make anything intrinsically more efficient, and they come with overhead.

The simple answer is just to be efficient about processing each entity in the game.

• Don't process every entity each frame.
• Split processing into stuff that needs doing often and stuff that does not need doing often.
• Spread long-running algorithms across multiple frames so that they don't block processing in other systems.
• Ensure that efficient algorithms and data structures are used.
• Cache the results of expensive calculations if they are likely to be used again.
• +1 for this, for actually explaining what's going on, rather than wittering on about graphics like me. :) Jul 23 '12 at 13:33
• To be fair, I gave you +1 too because I forgot about that aspect, and it's very true - take gfx out of the equation and it's like making computers 2x or 3x faster instantly. Jul 23 '12 at 13:36
• I've heard that DF is now multithreaded, but at least until recently, it was all done in a single thread. (It may still be, not sure) Jul 23 '12 at 20:11
• @MooingDuck It still isn't. Jul 24 '12 at 10:01

Dwarf Fortress is not open source, and while there is a lot of conjecture and reverse engineering that can go into how that all works, I will instead focus on some basic techniques for optimizing a 3D (not 3D graphics, 3D world) roguelike of the same type.

As is the case with all video games there are a lot of smoke and mirrors that are creating the illusion of complexity from simple rules and systems. These range from using simple random numbers for aimless movement all the way out to pre-baking a high-level mesh of nodes for pathfinding.

### Movement

Speaking of pathfinding, this can often be a very difficult problem to solve for large spaces like a DF map can be (as much as 768x768x64 IIRC) however the problem can be simplified and sped up in these ways:

• Pre-baked node network: When the map is created the world can be broken into chunks, and each chunk can have its exits and entrances mapped. When a chunk is updated, such as when a wall is built, only that chunk's network needs to be updated.
• Staged Pathfinding: Running a path over the whole map, cell for cell, would take a lot of time, instead you would pathfind over the larger chunk network, which maps all the connection between chunks, and then only run an intra-chunk path when moving from chunk to chunk. This does 2 things for you. It speeds up the pathfinding by breaking it up over many smaller pieces, and it also enables the unit to change direction part way along the path when a chunk updates. It would re-path across the large network if any of the nodes it needs to cross update.
• Random steering: When not moving to a goal, the unit need only walk aimlessly. Many roguelikes just move the unit in a random direction, which feels unnatural. Various steering techniques can be used, simple ones are favoring to move in a straight line and having less and less chance to move in the directions radiating to the back, which would have only about a 1% chance. So the unit would sometimes reverse direction entirely, but only rarely.

I won't cover the basics of pathfinding. Most roguelikes use A*, but there are other methods to skin the cat. Mmmm Cat leather..

One of the major things that makes DF units pop and feel alive is their personal goal list. In truth many roguelike games have this on a basic level. Essentially each unit has a list of desires (and for your dorfs, tasks they might pick up that you're asking to be done) and they will pick from them based on their personality (stats.)

Some tasks have requirements. Making a leather skirt requires the dorf be in such-and-such a shop that has X items. So those all are checked and added as tasks to their list. Simple as that.

Since most of the time a unit will be in transit, the checks for what units are doing can be very fast, only a few units will be making a choice at any given moment, and so overall there is no slow down even for hundreds or thousands of units. And remember, in DF everything from bees to troglodytes to trees are units.

In doing some additional research it is clear that, hilariously, DF is doing a terrible job of pathfinding in general. It isn't breaking the map into chunks, it is breaking the map into segments or areas that are connected, (which is better than nothing for sure) so my above assessment is even less an example of how DF works than I thought. :) Which is not to say that DF is anything less than amazing for a million other reasons.

It goes to show that what is important in a game is the game play. Not graphics, not great programing, not great writing, not great music, not even the interface; nothing else is even 1% as important as the game itself.

• 1024X1024X32? Vertical is 128 or 256 I believe. Its far bigger than 32. Jul 24 '12 at 3:09
• @Lawton I believe I am incorrect about the the max size, but the height is not that incorrect. I'm correcting it. I loaded the game and the embark I am on is only about 45 levels high. More might be 'simulated' but I don't have access to them for digging or building. Jul 24 '12 at 4:49
• @DampeS8N I just loaded up an unmodded, medium map and was able to view from +16 to -156, around 180 levels. 40 level maps are the exception, not the rule. Even if you are only able to dig 40 of them due to being blocked by an aquifer or lava, its not relevant, because the area underneath is still populated with simulated fun. Jul 24 '12 at 13:03
• @Lawton, my map only allows me to page through 45 levels. They are absolutely variable but I wasn't easily able to find numbers on how many each map has accessible. It also may depend on what version of the game we are using. In the end it isn't all that important for my answer. Jul 24 '12 at 13:05
• It's an old post but depths in dwarf fortress depend on sedimentary layers. Like the earth crust is thicker on some places. DF is not completely geological correct but man they studies it like pro's :D. Dec 6 '15 at 0:54

Well [the pathfinding] looks amazing from my end, since there's a metric ton of characters all doing it at once.

TA: The dwarves themselves mostly move around with A*, with the regular old street-distance heuristic. The tricky part is that it can't really call A* if they don't know they can get there in advance, or it'll end up flooding the map and killing the processor.

I don't know if this is definitely how he prevents "flooding the map", but the usual way to do this in games is using an alteration to A* called Hierarchical Path-Finding A*, or HPA*. The idea is to break the grid into many fewer but larger chunks, then use A* to find the best path from each chunk to its neighboring chunks. You can use this to build a much smaller graph over which to run A* for each unit over.

You can also group those chunks into even larger chunks, which is where the "hierarchical" comes from.

This algorithm only finds near-optimal paths, but for games like Dwarf-Fortress that is usually ok. It is still guaranteed to find a path if one exists; and when there is no path, it will only flood-fill the smaller graph, saving an enormous amount of time.

There is also an abstraction of HPA* that deals with units which can go over some terrain but not others (such as cliffs, which air-units can traverse over but ground-units can't). It's called HAA*, and there is a very accessible article explaining it here.

• I found this video from the RimWorld developer very enlightening. It's a similar game, while only in 2D. He explains the basics behind pathfinding and the benefits of separating the map into regions. youtube.com/watch?v=RMBQn_sg7DA
– Sire
Nov 5 '14 at 12:00

If anything, it's the opposite - the whole thing runs on one thread and it's now hitting the point where that is becoming the blocking factor (last time I checked!)

The reason it's fast is that there's no fancy graphics. It's deceptive, but the main thing that slows stuff down is drawing things (think upwards of two thirds of a frame in AAA titles). Since dwarf fortress is quite basic, it devotes the rest of that time to doing interesting things.

• One data point in favour of this is the OpenGL rewrite from a while back which I remember brought a massive frame rate increase. So even doing the simple sprite graphics that DF has efficiently made an impact. Jul 23 '12 at 15:13
• +1 Strip graphics = massive amount of spare time for gameplay computing Jul 23 '12 at 16:52
• It is worth noting that DF's graphics are about as demanding as any modern indie 2D game. There are many textures, even if they are small and simple, and they can be spread over full HD monitors of real-estate. There aren't any flashy particle effects or what have you, but the raw "paint all these pixels" work is still present and due to the number of draw calls needed for the tiny tile sizes is actually a pretty big challenge to make performant. Apr 28 '14 at 19:11
• It may look like a silly answer at first, but this is correct, imagine what can be done with 4-12gb gram and 4-12tflops GPU power if you don't need to draw complex primitives, unwraps, textures, transform 3d vectors into a 2d array of pixels, shaders, etc Feb 19 '18 at 22:30

I don't know how DF is coded but the amount of AIs doesn't really impress me because what people often oversee it that AI doesn't need precision. It's perfectly viable to do most stuff only every few seconds. It's also viable to use imprecise calculations. Imperfection saves a lot performance. You can run the decision making routine of 100 units every 100 ms, or you can run it for 1000 units every second, it will take the same amount of CPU time but well... you have 10 times the units.

Here is a simple example how one could handle a lot units:

int curUnit;
Array<Unit> units;
[...]
while([...]) // Game Loop
{
[...game logic...]
// process 10 AIs per Frame
for(int i=0; i++; i<10)
{
curUnit++
if(curUnit == units.length()) curUnit=0;
if(curUnit < units.length())
units[curUnit].processAI();
}

// Update the position of all units, this should be as optimized as possible
foreach(Unit& unit in units){ unit.move(); };

[...graphics...]
}


The AI will become less and less responsive the more there are, but the player probably will only notice it in extreme cases.

• There is a lot to be said for stepping through stacks of tasks that are not frame-level precise. AAA games often explicitly time-box each individual department in percentages of frames so that one department can't step on the toes of other departments. If the method for animating tree sway based on wind speed and direction is slow, fewer trees will be processed rather than eating up the budget of other teams. Apr 28 '14 at 19:04