How to automatically test a game's mechanics without duplicating work?

Question

I've heard and read enough programmers firmly advocating automatic tests. According to many, tests are themselves part of a code's functionality, untested code is broken and/or legacy by definition, long-term manual testing is more time-consuming and provides far weaker guarantees against failures than automatic testing... etc.

I'm trying to hobbystically develop a turn-based Pokemon-like game. While talking to a far more experienced developer I said that when I add a new attack I run my game, go into 1v1 combat, use this move and see if it does what its supposed to do. His answer was: Why won't you instead write a test that goes into a 1v1 combat, uses this move and checks if it does what it's supposed to do? Doing what you're doing manually may be faster than writing tests if you do it once, twice, thrice. But after 10 times? One hundred times?

I'm thinking about what he said. I'm thinking and I still don't feel convinced. About any other kind of project, perhaps. But a game?

Assume I'm adding lifesteal to a monster's attack. Problem is I'll have to anyway boot my game and play it! To see if it feels right, if it plays right, if not for anything else.

Tests on the other hand add maintenance cost: for example, once I (for balancement reasons) change the lifesteal coefficient, I'll have to reflect that change in tests. (Or I won't have to reflect this change in tests if I copy&paste the formula, parametrized by attack strength, lifesteal coefficient, etc, but doesn't this defeat the purpose of testing?)

Automatic tests are supposed to replace manual tests; but given I have to manually play my game anyway, isn't this duplication of work?

Or is my thinking wrong? What am I missing?

Answer 1

Testing is an investment in your future. While an individual test might duplicate some aspect of manual testing you are about to do in any given run of the game, a robust suite of tests can in the long run cover far more scenarios than that small bit of overlapping work (manual testing can also more effectively test more complex scenarios where writing automatic tests might be too difficult).

Writing tests is an overhead, yes. So is maintaining them when the things you are testing change. It's also much harder to retrofit tests onto a big project not built from scratch with that goal in mind, as there are some technical design choices that are less compatible with easy, isolated testing than others.

Whether or not that overhead is worth the benefit depends on you and the scale and scope of your project.

The payoff is that automated tests can be run automatically and require far, far less work on your part that equivalent manual testing coverage would. In your example, sure, you manually run the game and ad-hoc test your change.

But do you do it on every platform you are shipping on? On every build configuration? Automated tests can do so easily, which means you can catch errors that only manifest on one or two platforms/configurations that you don't regularly test more quickly.

Similarly, while you may ad-hoc test that one ability change, do you manually go back and ad-hoc test every other ability? Probably not, because you're operating under the assumption that you didn't make a change that could impact those other abilities. But you are a programmer, and therefore, you make mistakes ("bugs") and therefor you could have accidentally made your change in a such a way as to have unintentionally broke something you were not expecting. An automated test suite could catch that.

Granted, the value of the test suite is dependent on what you test and how, and there is a point of diminishing returns in the investment (100% test coverage is often impractical, for example).

For example, it's probably worthwhile to write a test to validate the results of a math function that presumes a left- or right-handed convention, as change in that function to prefer the other convention will probably destabilize a lot and should be flagged. But a test for the outcome of some damage calculation may not be a good candidate for a test, at least not early on when you are iterating rapidly on game balance. That sort of test is perhaps best added later, once balance has hardened a little, to alert you that you may have made a change that has balance implications.

Answer 2

First off, 100% test coverage is not a realistic goal, and games especially tend to incorporate elements that are difficult to test reliably, for example when aspects like timing, physics or (nontrivial) AI become involved. In addition, automated tests are neither infallible nor by definition superior to manual testing, so your friends' stance appears a little... overzealous to me.

Howver, that does not mean that games can't massively benefit from automated tests. A common approach in test automation is to prioritize areas that would benefit the most and work your way down the list until you reach a point where the efforts outweigh the advantages. Typical criteria are:

• How often does this code change / how likely is it to introduce a bug here?
• How severe are the consequences if it breaks?
• How easy is it to test?
• How obvious (or not) would a bug be during manual tests?

I'd recommend the following starting points:

Sanity checks

If game stats and actions depend on complicated and/or frequently tweaked formulae or functions, it's rarely necessary to check for minor deviations (unless your game is really balanced on a knife's edge), and updating the test cases every time can take a lot of work. What you do want to guard against, however, are gamebreaking numbers and unexpected interactions. Say, an ability that does 20 times the expected damage on a critical hit, but only against a specific enemy type.

Instead of setting up a test case that checks the exact damage output and secondary effects of every attack, I would write a test that goes through all attacks and checks if the numbers are within a specific range that's appropriate for their power level. Tweaking an ability by 10% for balance reasons shouldn't cause any tests to fail, but extreme outliers (which are probably unintended) should.

Core rules and algorithms

There will probably be pieces of code (say, the save/load mechanism, network code, pathfinding, core game rules...) that are critical to your game working as intended. They are also likely to interact with a lot of different modules, and thus be easily affected by changes to those. You might, for example, introduce a new enemy type, forget to serialize one of their stats and mess up the savegame format, which only becomes apparent during a longer playthrough, which you can't do after every little change.

For these central elements, automated tests are usually worth the effort because you absolutely don't want them to break.

Rapid iteration

Few finished games look anything like their early versions. You will probably make some sweeping changes that affect most of the existing content in one way or another, for example by adding a new stat or game mechanic. This, essentially, resets the "tested and mature" status of all your code and assets and the amount of extra testing required will grow with the size of your game.

You can't just skip manual tests, of course. There's no replacement for actually playing your game (and having others play it as well). But having a battery of tests ready to go, even if they're imperfect and incomplete, can allow you to vet and tweak these sweeping changes much faster, and find many inconsistencies the second they are introduced.

You're not limited to testing for correctness, by the way. If you're worried about balance, a scipt that runs a battery of different combinations, items, matchups or whatever can yield a lot more valuable (and reliable) data points than a handful of manual tests.

Answer 3

Problem is I'll have to anyway boot my game and play it! To see if it feels right, if it plays right, if not for anything else.

Yes, you need to do that while you are iterating on that specific situation you are implementing right now. But think ahead a couple years in the future. You might be working on a completely different aspect of the game and accidentally break this situation you though you had done. Are you going to test every single thing in your game after every change? Certainly not manually all by yourself. So how long will it take you to notice the bug? Will you then be able to easily connect it to that specific change you made? But if you have an automated test suit, you can easily test your entire codebase after every change.

Tests on the other hand add maintenance cost: for example, once I (for balancement reasons) change the lifesteal coefficient, I'll have to reflect that change in tests.

Yes, and that's a good thing, because now you have to be aware of every ramification of your balance change.

Does the tutorial still play out the way you intended? Is that one boss fight still winnable with the intended strategy? Is that other boss still immune to life steal? In order to detect those problems you might have to play through your whole game from start to finish. But an automated test can tell you within seconds to minutes.

So an automated test allows you to go through every situation where the outcome is affected and confirm one by one that this is still the outcome you intended.

Automatic tests are supposed to replace manual tests; but given I have to manually play my game anyway, isn't this duplication of work?

Automatic tests are not supposed to replace manual tests. They are supposed to augment them. While it can not replace the "how does it feel?" playtesting, it can save you a ton of work in regression testing (testing again and again if the things which used to work still work).