Are there methods of automated testing of games?
Specific experiences are appreciated, with relevant information about the project such as platform and type of game if that helps with clarification.
Game Development Stack Exchange is a question and answer site for professional and independent game developers. It only takes a minute to sign up.Sign up to join this community
One-person independent game. It was a multiplayer tank game with destructible terrain, and the destructible terrain and collision code proved somewhat flaky.
I ended up rigging up some basic dumb AIs (by "dumb", I mean "absolutely idiotic" - they would randomly choose "drive towards an enemy tank", "drive away from an enemy tank", and "drive in a random direction", while randomly firing the main weapon) and playing the game at maximum framerate while recording keypresses. I got about 10-15x realtime. The code was heavily asserted, so if anything went wrong, it would dump the entire keypress log to disk along with an error report and the initial random seed. I could then go and replay the keypress log to duplicate the state exactly, or just debug from the error report.
I left it running constantly for literally months. At the beginning it would rarely get an hour without crashing - I had to sit there and babysit it for a week, killing several obscure bugs per day. Eventually it got to the point where it was running for a week between failures, which translates to about 1500 player hours per crash.
It was invaluable and I heartily recommend it.
For an MMO I worked on (100ish developers, PC focused), we tried to add a huge variety of automated testing with varying success. Here's what worked:
What didn't work:
Working on a 4x strategy game with 3d combat (think Homeworld meets Masters Of Orion) that unfortunately never saw the light of day as the company ran out of funding..
I always ensured that you could play the game without human players so we could leave the game running overnight.
We could turn off the 3d combat (simplified to a random result) and we left the AI strategy engine playing itself. This found numerous bugs and issues. Not only show stopper bugs but strategy bugs where the (eg) AI strategies would get deadlocked and spend 1000s of turns not doing "the right thing". These sort of bugs were difficult to spot just "playing the game".
On a first person shooter I worked on (Descent 3 -- linux/mac/windows, ~30 people on the team in 1999), the demo recording/playback capability turned out to be extremely useful. I made an option where you could playback the demo as fast as the game could render frames, and that became a great way to verify performance after a bunch of things changed.
It also exercised a lot of the code beyond the rendering system, so it was a nice sanity check. After making a bunch of changes I could just run the demo playback of 10 minutes of gameplay. Many times it would catch a bug in an area I wouldn't have thought to check myself.
We had a openworld shooter (x360,PS3,PC) that used a quick smoketest on the build server - it loaded the game, stepped through the front end, ran [the avatar] forwards, dumped a screenshot, and exited. If cctray detected the clean exit the build was a success.
We ran it for about the last year of the project, and with a team size of ~100 devs.
It was effective at catching showstopping bugs but it was easy to create a build that passed the smoketest but failed most "real" levels, or didn't work in multiplayer, or nobbled the AI, so it wasn't perfect. It was definitely worth doing.
I've heard since I left they've started running a larger range of smoketests, farmed out to multiple PCs. Apparently maintaining the smoketests is an issue, and there's a small team dedicated to just keeping the build servers and software maintained, so I can't say if that's been a success or not.
My experience with Automated Testing during the development of Crysis 2 is available here: http://yetanothergameprogrammingblog.blogspot.com/2010/06/aaa-automated-testing.html
Game development is actually one of those cases where unit testing seems to make some sense to me, because the interactions between discrete systems are so common. Design-by-contract is of course a part of this, and should be planned on from day one of development, but I don't see why it couldn't be implemented later assuming the wherewithal to do so exists.
The hard part is, of course, integration testing. Lots of a game can be tested just by demo-looping it or something, but that stuff is conceptually fairly easy to debug--where I'd be more interested in spending my time is exposing bugs that will happen when a player does something, with the mindset that a bug the player never sees is obviously less important than a bug the player does.
Which is pretty difficult, obviously. Tactics that work on other applications (fuzzing, expected-pass/expected-fail, etc.) don't work so well here. In scriptable systems it seems like building a test set of scripts to simulate a player are the way to go (see JZig's answer). But testing specifically for stuff a player may encounter directly strikes me as the best place to focus your time for both human and automated testing purposes.