Game testing using mock input

Question

In this post Patrick Hughes suggested, and michael.bartnett was kind to explain a bit how to do that. While it seems rather clear how to implement and use it, I'm still a bit doubt if this useful enough. Does this kind of "testing" is used widely? I mean every change in a game usually change either how input is processed or a response to same input. This makes all recorded input obsolete. Or maybe I'm missing something?

I feel like implementing a unit testing for core parts (see also) of a game are much more useful, despite it also seems much harder to implement.

So the question is, does mocking input is really good at something in game testing?

Answer 1

Record+playback in a complex, interactive environment like a game pretty much has to be done in situ and not as part of a traditional unit test. The recording is intended to help QA set up a saved game and give a repeatable scenario to engineering for bug fixing.

Unit testing of core components IS useful, but only in the context of ensuring that the engine itself is functioning properly.

Examples of engine unit tests would be to load a static scene and compare a render versus renders in the past, or (more on topic) to load a player and feed it recorded inputs to test whether it ends up at the same coordinates as previous runs.

There are other, non-deterministic tests that are very useful in online games. The best example is load testing with a large number of headless clients fed from a pool of recorded inputs, this would catch networking problems and help the engineers optimize.

And there are failure tests where all sorts of bad juju is thrown at the engine to help flush out problems, including malformed inputs as part of the test.

In summary, record+playback is useful but in constrained ways that supplement living people instead of replacing them.