Automated testing of games [closed]

Question

Are there methods of automated testing of games?

Specific experiences are appreciated, with relevant information about the project such as platform and type of game if that helps with clarification.

Answer 1

One-person independent game. It was a multiplayer tank game with destructible terrain, and the destructible terrain and collision code proved somewhat flaky.

I ended up rigging up some basic dumb AIs (by "dumb", I mean "absolutely idiotic" - they would randomly choose "drive towards an enemy tank", "drive away from an enemy tank", and "drive in a random direction", while randomly firing the main weapon) and playing the game at maximum framerate while recording keypresses. I got about 10-15x realtime. The code was heavily asserted, so if anything went wrong, it would dump the entire keypress log to disk along with an error report and the initial random seed. I could then go and replay the keypress log to duplicate the state exactly, or just debug from the error report.

I left it running constantly for literally months. At the beginning it would rarely get an hour without crashing - I had to sit there and babysit it for a week, killing several obscure bugs per day. Eventually it got to the point where it was running for a week between failures, which translates to about 1500 player hours per crash.

It was invaluable and I heartily recommend it.

Answer 2

For an MMO I worked on (100ish developers, PC focused), we tried to add a huge variety of automated testing with varying success. Here's what worked:

• Basic tests during our automated build process were a huge win. This included tasks such as creating a character, transferring maps, running some scriptable UI tests and looking for expected behavior. This caught a huge number of bugs before they actually got to the rest of the company.
• On the server infrastructure end, we developed a bunch of different automated tests that simulated typical MMO server transactions. We could then play these back under a variety of circumstances to compare performance or ensure safety. Over time these tests became more and more accurate until it turned into a playback of live recorded data
• We wrote a "fake player" that would randomly roam around the world, jump, kill things, and say random things in chat. This found a huge number of physics and infrastructure issues.

What didn't work:

• We tried to add some very specific combat-oriented automated tests to the automated builder, but this basically never worked. It would work for about 3 days after being implemented, until a designer or artist changed something and the test would fail, throwing off the build failed alarms. 90% of the time it wasn't a real problem. These tests were too fragile, and actually testing specific gameplay on a specific map with specific powers can be unmaintainable
• We tried implementing an automated performance test that would compare client performance (average FPS, etc) against recorded performance from a week ago. This was also fairly fragile as the demos we used for this tended to rot fairly often, and it was difficult to ascertain if a slowdown was caused by an actual loss in performance or some side effect of the test process.

Answer 3

Working on a 4x strategy game with 3d combat (think Homeworld meets Masters Of Orion) that unfortunately never saw the light of day as the company ran out of funding..

I always ensured that you could play the game without human players so we could leave the game running overnight.

We could turn off the 3d combat (simplified to a random result) and we left the AI strategy engine playing itself. This found numerous bugs and issues. Not only show stopper bugs but strategy bugs where the (eg) AI strategies would get deadlocked and spend 1000s of turns not doing "the right thing". These sort of bugs were difficult to spot just "playing the game".

Answer 4

On a first person shooter I worked on (Descent 3 -- linux/mac/windows, ~30 people on the team in 1999), the demo recording/playback capability turned out to be extremely useful. I made an option where you could playback the demo as fast as the game could render frames, and that became a great way to verify performance after a bunch of things changed.

It also exercised a lot of the code beyond the rendering system, so it was a nice sanity check. After making a bunch of changes I could just run the demo playback of 10 minutes of gameplay. Many times it would catch a bug in an area I wouldn't have thought to check myself.

Answer 5

We had a openworld shooter (x360,PS3,PC) that used a quick smoketest on the build server - it loaded the game, stepped through the front end, ran [the avatar] forwards, dumped a screenshot, and exited. If cctray detected the clean exit the build was a success.

We ran it for about the last year of the project, and with a team size of ~100 devs.

It was effective at catching showstopping bugs but it was easy to create a build that passed the smoketest but failed most "real" levels, or didn't work in multiplayer, or nobbled the AI, so it wasn't perfect. It was definitely worth doing.

I've heard since I left they've started running a larger range of smoketests, farmed out to multiple PCs. Apparently maintaining the smoketests is an issue, and there's a small team dedicated to just keeping the build servers and software maintained, so I can't say if that's been a success or not.

Answer 6

My experience with Automated Testing during the development of Crysis 2 is available here: http://yetanothergameprogrammingblog.blogspot.com/2010/06/aaa-automated-testing.html

Summary:

• Automated testing improved deliverables stability, increasing productivity for both content creators' and engineers
• Automated testing is an effective tool to improve code quality and reduce the chances of having to work overtime
• The Game Industry as a whole is very reactionary in general, automated testing meets several irrational arguments against
• Don't call it testing, call it something else, almost anything else (Look at Behavior-Driven-Development)
• Be flexible, writing good tests is hard and require skills that are not widely available in the Game Industry

Answer 7

Game development is actually one of those cases where unit testing seems to make some sense to me, because the interactions between discrete systems are so common. Design-by-contract is of course a part of this, and should be planned on from day one of development, but I don't see why it couldn't be implemented later assuming the wherewithal to do so exists.

The hard part is, of course, integration testing. Lots of a game can be tested just by demo-looping it or something, but that stuff is conceptually fairly easy to debug--where I'd be more interested in spending my time is exposing bugs that will happen when a player does something, with the mindset that a bug the player never sees is obviously less important than a bug the player does.

Which is pretty difficult, obviously. Tactics that work on other applications (fuzzing, expected-pass/expected-fail, etc.) don't work so well here. In scriptable systems it seems like building a test set of scripts to simulate a player are the way to go (see JZig's answer). But testing specifically for stuff a player may encounter directly strikes me as the best place to focus your time for both human and automated testing purposes.