Value of describing game mechanics in a data file instead of hardcoding?

Question

I am working on a turn-based, dice roll-based roleplaying game simulation engine. Here are a few relevant bits about it:

Background

1. The system is d20 based. The complexity of the system is somewhat less than well-known RPG systems like Pathfinder or D&D, but not so simple as to make it trivial. As an example, we have 3 primary stats (hitpoints, stamina and mana), 3 secondary stats, 3 classes, each with 3 specialization options (pick one), and a total of 30 unique abilities, some active (player spends their turn to do it), some reactive (occurs at the player's option after a certain condition is met, possibly outside of their normal turn), and some passive (always in effect). Players get to pick certain abilities to "equip" into their character, based on their rank and progression through our advancement system.

2. There is a group of people in my MMO guild who maintain the design of the system. They issue "patches" a handful of times per year. The scope of a patch can be fundamental, like introducing an entirely new system (rare; yearly or less often), or minor, like changing the name of an ability, or increasing/decreasing the numerical resource cost or effect of an ability, like adjusting the maximum amount of hitpoints you can heal with one heal spell.

3. The cadence and scope of the patches is outside my control, with me being the designer of the simulation engine. A patch may require a lot of work for a program to support a new system or major change to the rules, or it may just be changing a number or a string somewhere.

4. My goal is to get a core "engine" built that fully models the mechanics of the system, then have several components leveraging it:

4(a). A Monte Carlo simulator, that generates pseudo-random battles based on tunable criteria; for example, simulate 1,000,000 1v1 fights between a Warrior and a Mage.

4(b). A persistent database of all our characters with their up to date "builds" along with all the metadata associated; e.g., name, race/species, abilities slotted, max HP/SP/MP, rank and progression boolean flags, etc.

4(c). A web app providing a user interface to the database and a "live" combat simulator, where players can make choices according to the rules of the system, are restricted from taking actions that violate the rules, but can effectively play out a fight, either PvP (against other players) or PvE (against NPC enemies created by a game master). Players can do things like use abilities, attack, heal, or attempt to retreat from battle. NPCs are given configurable character sheets that can have custom-designed abilities above and beyond what players can normally do.

Describing Combat System Patch Trends

In this system, there are certain "near-constants" that are unlikely to ever change; or if they do, it's only once or twice a decade. I'm really not worried about them changing, because if they do, I will simply accept the development cost of implementing them, even if I have to do a lot of rework to existing code. Examples: our core dice roll is based on a d20. Our core stats are hitpoints, stamina points and mana points. Our three classes are Warrior, Rogue and Mage.

Then, there are less constant things, that will change, on average, about once a year. Examples include the list of abilities -- we may add or remove entire abilities, or completely change the mechanics of an ability for balance's sake. It may be possible to describe these changes in terms of general purpose rules or fundamental operations; for example, you could build up an ability's effects out of a grab bag of predefined primitives, like "deals damage," "heals a given target," "confers a bonus to attack rolls for X rounds," and so on.

And at the bottom of the pyramid, there are tweaks, which are just extremely isolated changes that happen somewhat frequently, but are easiest to describe in code, like the number of points of damage a certain attack ability deals. I'd like to be able to implement these very quickly and easily ("DRY" if possible) and have all the downstream tools/programs adopt the new change without any other code changes to them.

My Questions

How much of the system should I directly hard-code using the programming language I'm planning for my project (probably Zig)? How much, if anything, should I encode in a data format, like JSON?

I've looked at the design of larger, more complex games before while modding, like Sins of a Solar Empire, Skyrim and Starcraft 2. I notice that many major game designers build their game systems based on the following architecture:

• Certain basic attributes of their game system are hard-coded into the game code. A modder, for example, is unable to change these aspects of the game in closed-source games. In my case, if the system maintainers decide to change these aspects of the game, I would have to change my code. That's OK with me, as long as I anticipate these things and make it so the code changes can be small and infrequent most of the time. Frequent, major code churn is undesirable for obvious reasons.
• They create fundamental functions that perform tunable, primitive operations, then have some kind of data definition files that describe most of the game content and mechanics. User-visible skills/effects/weapons/etc. have their mechanics described as a series of these primitives. A closed-source game is "moddable" by applying these primitives in different ways to adjust the behavior of existing game assets or abilities (weapons, items, skills, ...) or even creating entire new ones.

For a game system whose code will be 100% open source on GitHub, is there any reason to define game mechanics in data files as opposed to just hard-coding it all? Do games only create these extensible elements for modding purposes for the community, or are there maintainability advantages of doing this?

Example: Sins of a Solar Empire defines most of its ships and abilities in terms of JSON files. These JSON files will define, say, the list of individual weapon mounts equipped on a ship class -- maybe it has 3 lasers and 2 missile launchers. When the game starts up, it reads all the JSON files and builds a representation of the full game's rules dynamically in memory by creating data structures at runtime for each of the ship classes, weapon types, etc. defined in JSON.

Obviously, going with this dynamic, data-driven approach is much slower than having native code directly describe the behavior of the system. Performance-wise, hard-coding wins, hands down (both for loading time and for actually executing the simulation of the game mechanics). With Zig, I may be able to chop down much of the loading time using comptime, but there is still going to be a cost there.

Is there a maintainability advantage to defining content/systems/abilities/items/etc. in data files as opposed to hard-coding them? Does it take more work to update these things in code and then propagate them through the whole stack, as opposed to loading them dynamically from data files?

Are there any best practices to figure out where to make that division or layering separation between things that are hard-coded and things that are data-defined?

Why I'm asking

My biggest "fear" is that I'll complete the implementation of everything, and then a patch will come along that will necessitate major re-work in the database, and the core engine, and the UI, and the network code (API) between client/server.

I'm certain that some changes to the system will make this absolutely unavoidable, and I've accepted that reality. But how do I minimize the frequency of having to do those heavy changes, and ideally just be able to adjust things in one place? It feels like using data files to describe system mechanics will help with that a lot. But I'm still not sure exactly what data to define to achieve a good balance between flexibility and expressiveness.

I know that taking this idea to the other extreme ends up producing something like the Enterprise Rules Engine anti-pattern (inner platform effect), but hard-coding everything also feels like a foolish design.

Answer 1

The problem with building an architecture that is optimized for making certain predicted changes is that such an architecture usually makes unpredicted changes more complicated. Which is a problem because it is usually very difficult to predict if, when and how a system will be modified in the future. This applies to games in particular, where you often "find the fun" by repeatedly playtesting and iterating on your design. So your prediction on patch trends in this question could very likely turn out to be very wrong.

Far too often in my life did I design convoluted architectures that would have made certain changes really easy in the future, just for those changes to never happen. Turned out that the designers and/or users of the system had far different ideas what things they wanted to change or add. So my super-flexible-super-extensible architecture remained unchanged, while I had to do ugly workarounds and painful refactorings to fulfill requirements I had never expected.

Over time I switched to a different approach:

1. In the prototyping stage, I keep it simple and stupid. Often the simple solutions are the best solutions. The fewer layers of abstraction you have, the fewer layers of abstraction need to change when you figure out that your abstraction was misguided. And the fewer hours of work you end up losing when it turns out that the architecture you over-engineered actually wasn't needed at all.

   So when you are just starting out, shamelessly hardcode everything. Changing the attack of the short sword from 10 to 12 requires a recompile? So what? The code-base is still small, so recompiling is quick.

2. When I have a difficult change, I refactor to make the difficult change easy, then make the easy change (quote from Kent Beck). The problem with most KISS architectures is that as soon as you start iterating on them, you will encounter change requests which take a lot of work at a lot of different places. This is when I start engineering abstraction layers, not before. It ensures that I don't start over-engineering before I have at least one confirmed use-case for a certain abstraction.

   For example, when I discover that compilation times get longer and we keep changing weapon stats over and over again, then it might be time to move all those weapon stats to a configuration file. But the good thing is that at that point, we already have some experience with the system. I know what properties the weapons have, and I probably have a much better idea of what requirements that configuration file actually needs to fulfill. So I have far more data to make my design decisions on.

Answer 2

You can start by having no data files. And by pursuing a single source of truth, your code might evolve to have sections where it calls a method multiple times in a row with different data for initialization. Which then can turn into some large data structure literal with all the data over which you loop over. And then it grows large enough - or changes often enough - that it makes sense to move it to its own file... And you might want to:

• Make sure it only has data.
• Allow non-programmers to tweak it.
• Use some other software to make it easier to work with it.

It is ultimately a judgement call to make it a JSON file or a CSV file or something else. There isn't a hard line because it is about making the workflow easier...

When you have few game components, and they are all significantly different from each other, and you do not need non-programmers to be able to tweak them, then you will find that using data files is more work than it is worth.

Conversely, if you have many game components, that differ only - or mostly - on data instead of mechanically, and you want to make it easier to edit them (either for development, or to make it easier to automatically test variations, or for designers to easily tweak the values), then data files are probably worth it.

---

For example, in some video game there could easily be hundreds - or thousands - of items. And this can make them hard to maintain if they are hardcoded.

And there could have - again, for example - 10 weapons, which differ only in how much damage they do, how much they cost in NPC shops, what sprite art they use, and stuff like that.

And you want designers to be able to quickly tweak these values for rapid prototyping.

Thus, you might use a CSV or JSON file instead.

---

Or, perhaps, you do not need to have designers tweak the values, but it might be convenient to use spreadsheet software to look at the values at a glance to have an idea of their progression, which might make a balance review easier. Plus, spreadsheet software will also make it easier to make data visualizations from the data.

And - similarly - spreadsheet software might allow you to use formulas to define these values quickly.

Or, simply, it is a matter of making it easier for you to add them to the game because a table is what the designers give you already. Or you might convince them to give spreadsheets to you. Tell them which is the table format, perhaps some example data, and they might express part of their future updates using them.

---

When you go over your list of components you are likely to find that they do not differ only on data. Which is arguably good, because this means you have differences of kind, not just of differences of degree.

Yet, you can create classes/systems for their behaviour. And then in your data file you specify which they use. For example a bow that inflicts poison damage would use the ranged weapon behaviour and the inflicts poison behavior, while differing from a crossbow that inflicts poison in data only.

Thus, these behaviours would be code.

---

Another reason to keep something in code is to have a guarantee that it is present. Usually you would design such that none of the components depend directly on another...

This goes both ways:

• Depending on data allows you to decouple some components of your game.

  For example, fire damage destroys wooden weapons, and corrosive damage destroys metal weapons. Then each weapon can specify their material. But at no point the code for fire damage has hardcoded the list of wooden weapons, and so on. In fact, there might be none in the game and the code still works.

• If a component is required to exist for some mechanics to work, it is better to keep it in code.

  For example, the aforementioned fire damage does not depend on any specific weapon, but it depends on some weapon type, having some material, and some way to destroy them. Thus, those are things you need to keep in code.

---

Now, let us say you have some vampiric powers in the game, which will steal hitpoints. Your data files can specify which attack has this effect, but for this effect to exist, hitpoints must exist.

Even if you generalize hitpoints, stamina and mana into a single concept, and let the power specify which one they work with. It does not mean that you will define hitpoints, stamina and mana from a data file (they are only three, they are mechanically different, and designers don't mess with them often).

However, there might come a time in the future when a data file becomes convenient (Perhaps in some future version there will be ten instead of three, they are more mechanically similar, and the designers are messing with them more often, so you decide to put them in a data file that they can tweak and test what happens instead of programming every change).

---

Thus, I want to reiterate that it is a judgement call.

Keep present requirements are paramount, so keep them the priority. And pay attention to the steps of your workflow that require most time and effort. Think how to aliviate them. And you might find that at some point data files would be a good option. Or something else entirely.

Answer 3

Obviously, going with this dynamic, data-driven approach is much slower than having native code directly describe the behavior of the system. Performance-wise, hard-coding wins, hands down (both for loading time and for actually executing the simulation of the game mechanics). With Zig, I may be able to chop down much of the loading time using comptime, but there is still going to be a cost there.

False dichotomy. You're comparing those as if it's about data-driven design vs hardcoding values inline, but so long as you put stuff into data structures, does it matter if those data structures are hardcoded static arrays in your classes, or are read in from a config file at the start? It's still data-driven.

And whether stuff is hardcoded into a datastructure in the code, or hardcoded as separate lines of algorithms, they're hardcoded, and suffer zero read-in time.

But as others have pointed out, the read-in time at program startup doesn't cause any significant time delay to reading in config files, compared to any kind of hardcoding.

So, what stuff should you put into data structures?

Everything you possibly can. The less code you write, the less code you have to maintain, and the more code reuse you have.

Let's take Furcadia as an example. When I started working on the codebase, every species of avatar was a separate branch in a switch statement, a separate function to call, and so on. Adding a new avatar was fairly simple: just search through all the client and server code for, say, "dragon", and copy each chunk of code for your new species. This had worked for decades, but the downside was, every new avatar meant a new client release.

I took all the various things that changed with each avatar (walk animation sequences, avatar ids, etc), and put them into an array, then made the code look up or loop over that array. This array was loaded in dynamically from the database on the serverside, and sent to the clientside on login and on change. Any art files the client lacked, it could request from the server.

That meant that adding a new avatar was just a case of uploading the new graphics to the server, and updating the database. No more massive client downloads, bigger with every avatar. No more client releases for cosmetic changes to the game. Not even a server restart. The system had become dynamic.

Even if, at the moment, your game is not networked, and won't have a server that will serve updates to the data, you might in future: and rather than update the whole client, it's way easier to just update data. That even means that you can dynamically update the client while the game is running, without a restart.

But even before you distribute the game, you'll likely see the benefits when it's time to debug. While you're testing, instead of quitting the application, editing the code, recompiling, restarting, and playing the game to get back to where you can reproduce the issue you were debugging... just change the data, reload it, and see your fix in the game immediately. So any time you can dynamically change something in the game rather than recompile, you save yourself hours of time in testing.

I'd go so far as to say that there is never an advantage to putting something inline in the code when you can put it in a data structure that the code can manipulate. And once you do that, it becomes super simple to move those data structures out to a config file, or even to a database like SQLite, making complex searches across the data massively simpler and faster.

JSON ain't bad for more complex data. CSV is ideal for simpler, tabular data, and makes migration to a database easier. Avoid XML, it's just crazy amounts of overhead.

Answer 4

A minor supplementary answer: many games help to separate mechanics from the engine with the use of a scripting language such as Lua. So for example, the core engine may be programmed in C++, but the logic for an ability might be coded in Lua. Often, the scripting language is interpreted rather than compiled, allowing game designers or modders to adjust existing mechanics or develop new mechanics without having to recompile the game.

The main disadvantage of an interpreted scripting language is that it's much slower than compiled code, so a well-designed game will try not to put any performance-critical code into the scripting language. How much the performance matters will completely depend on your game - for example, performance is typically much less of an issue in a turn-based game.

Example: Sins of a Solar Empire defines most of its ships and abilities in terms of JSON files. [...]

Obviously, going with this dynamic, data-driven approach is much slower than having native code directly describe the behavior of the system. Performance-wise, hard-coding wins, hands down (both for loading time and for actually executing the simulation of the game mechanics).

To be clear, pure data like what you would store in JSON is generally going to be loaded and parsed once, either when the game boots or the first time that data is accessed. There shouldn't be any performance impact during gameplay.

Is there a maintainability advantage to defining content/systems/abilities/items/etc. in data files as opposed to hard-coding them?

Absolutely. Here's a simple example: your team isn't sure about how the druid is balanced on paper. Half of the team thinks that the druid should be built one way, and the other half thinks the druid should be built a different way. You've hit the limits of what you can figure out through brainstorming, and want to playtest both builds of the druid to see which one works better.

If your druid's stats and abilities are defined in a configuration file, you can try out different druid settings by just swapping out the configuration file, without having to compile new builds (or having to make changes to the UI to support multiple hardcoded versions of the druid). You can save the different configuration files with different versions of the druid into a folder so that it's easy to refer back to them later. You can go back to an earlier version of the druid just by swapping the configuration file, without having to roll back the code in your version control system.

Answer 5

The project being described here sounds very similar to my Arena program for OSR-style classic D&D that I've maintained for about ten years now. Reflecting back on that, here's a few organizational tips I might share.

Anything in those kinds of game systems that gets presented in a table format or stat block is a sign of characteristics that are well-defined, vary often but in minimal ways, and are ripe for storage in a database format. For example in my case: the class levels, monsters, treasure types, saving throws, XP awards, and lots of similar stuff.

Things in the game system that are functional and presented as a paragraph block of text are less susceptible to this kind of abstraction. Usually any new kind of feature like this requires some custom code to implement. For example in my case, the monster database includes keywords for various special abilities, but each of those needed custom implementation code, and the code structure is a slightly unwieldy giant switch block (e.g., function makeSpecialAttack() in this file).

Similarly, a later addition was an implementation of many of the wizard attack spells in the original game -- stat-block things like level, range, and duration were obvious abstractions (here), but each spell needs custom code to simulate the text effect, and they way they get linked and used by each wizard character is a bit convoluted (here).

As you go forward, be sensitive to moments where you find yourself making minor changes to code and re-compiling over and over again as you iterate on certain experiments. (E.g.: If I want detailed printouts of two specific monsters in a fight, and I keep fiddling with two constants to identify which ones to focus on.) Or, if you start maintaining big initialization lists, and frequently modifying or extending them -- that's something that in the game would be presented as a table, and should be extracted to a data file.

Once you have a file-reading format and facility in your code, and you make a single object/database type that can read the stats from a file like that, you'll find that it's exceedingly easy to copy and modify that kind of code for new tables. Hopefully you get in the habit that when you see or think of a new table, the easiest thing to do will be to just instinctively implement that as a separate data file.

Briefly, among the many benefits are: (1) you can quickly modify data en masse without the delay of opening, editing, compiling code, (2) you can use spreadsheet tools to immediately view, sort, organize, highlight, compute stats on your data, (3) non-coders can view and tweak the data as desired, (4) the data can be exported, reported, and integrity-checked by other external tools (e.g., Github automatically displays .CSV files in a neat table format with headers), (5) the data can be easily moved and used in a completely different program (e.g., the monster database was re-used in my wilderness encounter simulator).

Answer 6

A data file is less expressive than code. Oddly enough, this can be an advantage, if you want to get things "right." There's a theorem in Computer Science called Rice's Theorem. It states that it is impossible to prove whether a semantic property is true for any arbitrary program. A semantic property describes what it does like "does it halt?" as opposed to syntactic properties like "does it have 'if' statements?" Most things people care about are semantic properties.

This means that if anybody questions whether your program is "right," it's a real challenge. You can't answer that in general, so you first have to demonstrate that your program is not just any arbitrary program - it has particular traits that let you answer some semantic properties about it. In your case, it would be things like "Is this rogue ability implemented properly."

Data files are less expressive than a Turing complete language, so it is typically much easier to demonstrate semantic properties like "correctness." I'm working on an application right now that chose to model a particular behavior as a state machine and a file of "constants" because we found it easier to prove the behaviors we care about from those two data-file inputs.

It's a balance. The less expressive the format, the easier it is to prove statements about it, but the less it can actually describe. Hardcoding is just one extreme end of that spectrum, with maximum expressiveness and minimal provability. Speaking in vague generalities, many programs find that some aspects are best described using a data file while others are better expressed via hardcoding.

There's two disadvantages to consider for datafiles:

• One has to define the engine that parses the files and acts upon them. This can be a challenging piece of code.
• When determining if any code change is valid, one has to consider any possible data file that could be provided, while if everything is hardcoded, one typically only has to prove that that one version works.

Data files should be leveraged when neither of those is a driving factor.

Answer 7

One of the most challenging aspects of game development is balancing. It is very difficult to compile, yet alone change, your code every time you increase damage of a single ability, only to increase it further a second later. This becomes even a bigger issue if you are using C++. C++ is easy to program (you should learn it fully first), and fast to run but there is one huge problem: compilation time.

Use data files load everything that is numeric, textual or enumeration based. When you have a data manager that can fill in your data structures, there will be next to no programming overhead using data files. Trying to offload logic is not easy nor a good idea most cases.

An additional suggestion, but a keyword during development that will reload data files instantly, this might take some time to ensure it will not cause any serious issues but it will worth the effort.

Answer 8

Data vs behavior

Example: Sins of a Solar Empire defines most of its ships and abilities in terms of JSON files. These JSON files will define, say, the list of individual weapon mounts equipped on a ship class

Obviously, going with this dynamic, data-driven approach is much slower than having native code directly describe the behavior of the system.

You're comparing apples to oranges. Or, more on topic, data to behavior.

Data is fantastic at changing over time. You can add things and change them long after the application was released. Behavior, however, is encoded statically in the application. Changing it requires a new release of the software.

When non-trivial, data is difficult to handle. This is because the developer cannot rely on the existence of specific data values, since data is inherently treated as an unknown value that will only be known during runtime, not development time.

Using your example of Sins of a Solar Empire, you will not find any logic in the game that relies on a ship having a specific amount of weapon mounts. It's likely only going to know what a weapon mount is, and that a ship has an unspecified variable amount of them.

---

Making your behavior data-driven

Given sufficient effort, you are able to turn anything that's currently static behavior into a data-driven format. However, this requires an additional layer of abstraction, as you're turning your concrete implementation into a configurable object with unknown values that you can't rely on during development time.

This kind of abstraction makes your application changeable post-release, but it comes at the cost of significant additional complexity. At some point, you have to balance how much complexity is too much, and whether the extra effort tax that this brings with it is worth the benefit that it renders to you. It also comes with some kind of risk assessment - do you want to pre-emptively make everything configurable, which takes a lot of effort, or do you want to be clever about which things you make configurable, effectively guessing and risking that you might not make something configurable that you will come to regret going forward?

There is no straightforward answer here. What you're asking to do is almost the very essence of what advanced/senior software developers engage in, in terms of technical analysis and software design.

This is highly contextual, on a case by case basis. It depends on your available resources for developer effort, the impact of features being unavailable because they were not pre-emptively developed, and your ability to accurately gauge in advance whether a feature is going to be needed or not.

Games that are open to modding tend to have invested extra time and effort in doing so, because they are banking on the moddability of their game benefiting them in the long run, either via increased long term player engagement or the devs' own ability to more easily test and patch the game. That investment is a gambit, and one that you need to assess for yourself.

Not to give this a negative tone, but games that release very similar new versions every year (typical examples include sports games and online shooters) tend not to spend any time on this because longevity of player engagement is not their goal (if anything, the opposite is more profitable for them).