3
\$\begingroup\$

For a hard 2D platformer I'm working on, I want to make an automated playtesting system. The goal is to allow every level to be tested in parallel as part of an automated test suite. These tests would both help to catch bugs and determine if a level is still possible to beat after any given change, which could both be huge time savers in a game that could take a few hours to finish.

How do other games approach this?

My initial approach has been to record some inputs I use to finish a level, and then replay those inputs in the test. Specifically, every frame I check if the input state is different from the last recorded input state, and if it is, save it along with a timestamp relative to when the player spawns into the level. Replays simply set the input state according to those timestamps.

This works maybe a quarter of the time. I suspect the issue is that small fluctuations in floating point math add up pretty quickly, and the input recordings make a lot of assumptions as to where the player is at any given frame which may not be true due to those deviations. (I also thought it might have to do with variable frame rate, but I didn't see any improvement from switching to FixedUpdate.)

I have a couple ideas for how to fix this, but they all have cons:

  1. Implement an AI. A fully autonomous agent is overkill; a more realistic alternative might take the input recordings and make small adjustments in real time, but that would still be a lot of work to make
  2. Fudge things. Record the state of every entity, and if it deviates enough during a replay, set it back to the recorded value. This smells bad to me - it feels like in this case, the behavior I'm testing is different enough from actual gameplay that the tests lose a lot of their value
  3. Explore deterministic physics. I've found approaches like this, but I'm not sure if they're relevant for my game - I only use Unity's physics for collisions (everything else is hand-rolled to avoid generic gamefeel), and I'm already loading the scene fresh at the start of each test, so I'm not sure what else could be achieved with a reload. I absolutely could be missing something, though

What else haven't I considered?

\$\endgroup\$
3
  • 1
    \$\begingroup\$ Instead of testing a full level, maybe it is possible to break it into segments that can be tested individually. Usually a puzzle platforme is built from those segments and it is enough to test once if you can cross the gap of X units with your double jump. Or go y distance between walls for upwards wall jump. This does not solve on how to run the tests but would make both building the level and testing easier \$\endgroup\$
    – Zibelas
    Commented Mar 29 at 8:11
  • \$\begingroup\$ Those segments could than be tested by something like a manual setup up docs.unity3d.com/Packages/[email protected]/manual/… that is fine tuned for the exact segment \$\endgroup\$
    – Zibelas
    Commented Mar 29 at 8:15
  • 1
    \$\begingroup\$ You may be interested in How to Make Insane, Procedural Platformer Levels and Procedural Level Design for Platform Games. Even though you're not trying to generate these levels procedurally, a common element in the methods these authors present is an automated feasibility checker. \$\endgroup\$
    – DMGregory
    Commented Mar 29 at 15:11

1 Answer 1

2
\$\begingroup\$

Automated testing of full levels is a difficult topic. As you've noticed, replaying inputs is tricky. Even if you get the timing right and switch to a deterministic physics engine, a small change in the level or gameplay code can make the replay invalid in various ways and require you to rerecord the level. Depending on how often this happens, you might find that the replay-testing system doesn't save you any time.

A recording also doesn't tell you everything and can even give you a false sense of security. For example, say that you adjust the collider for a platform incorrectly, making it possible to fall through the edge that looks like it should be solid. If you landed squarely on the middle of the platform when you recorded a playthrough, replaying the inputs won't help you realize that the collider doesn't line up with the edge.

Regarding #1, as an alternative to developing your own AI code from scratch, you can use an ML-driven system such as Unity's ML Agents package. In theory, with sufficient training, you'd end up with an agent that can play through any level if the level is beatable. Would this actually save you any time? That'll depend on you and your project. There are some of the same risks as replays - for example, if you change the player's jump height after training the agent, the agent might break down and need retraining.

Rather than immediately investing a lot of time into an automated testing system, you should start by analyzing your game and your development strategy and weighing the costs and benefits of automated testing.

Try to answer these questions:

  • How often am I expecting to make changes that could break some levels?
  • What types of changes could break levels? (E.g. adjusting the player's speed or jump height.)
  • How significant of an impact would each change have? For example, changing the player's jump height could have a drastic impact on all sections of all levels of a platformer, but changing the damage from an environmental hazard would only require you to retest areas where that hazard exists.
  • How many levels are there? How long would it take to re-test each level?

The goal here is to try to get a rough idea of how much time you would expect to spend manually retesting after changes, and determine whether developing an automated testing system would save more time than it costs.


A strategy recommendation

If you are expecting to be spending so much time doing regression testing (going back and testing something that used to work to see if you've broken it) that you really need an automated system, you might be approaching this from the wrong angle. You can save an enormous amount of time by making key decisions early and then building around the decisions that you've made.

For example, in a platformer, the player's speed and jump height should be determined as early as possible and then set in stone. Design levels around mechanics, rather than redesigning mechanics around levels.

Build levels one at a time in the order that they'll appear in the game, and try to get a level to a point where you're very happy with it before moving on to the next level. Then, do your best never to make a change that would break a level that you've already finished. When you introduce a new mechanic such as a new enemy, build a level with that new mechanic and get it to a point that you're happy with. During this process, you might refine the enemy behavior. Once the level is finished, however, you should keep that enemy as-is for all future levels. Again - design levels around mechanics, rather than redesigning mechanics around levels.

Any time you catch yourself wanting to change a mechanic that you've already used in previous levels, stop and really think about whether the change is justified. Were you happy with the mechanic in previous levels? Is the problem something you can solve by tweaking the level, rather than the mechanic? In some cases you can introduce a derivative or variant mechanic - for example, creating a new variant of an enemy rather than modifying the original enemy.

\$\endgroup\$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .