Take the 2-minute tour ×
Game Development Stack Exchange is a question and answer site for professional and independent game developers. It's 100% free, no registration required.

What are your experiences with automated testing of games? What worked, what didn't work?

Please mention what kind of game you were working on, what the target platforms were, the team size, project time span and any other relevant information about the project.

share|improve this question
add comment

closed as not constructive by bobobobo, Tetrad Apr 14 '13 at 5:32

As it currently stands, this question is not a good fit for our Q&A format. We expect answers to be supported by facts, references, or expertise, but this question will likely solicit debate, arguments, polling, or extended discussion. If you feel that this question can be improved and possibly reopened, visit the help center for guidance.If this question can be reworded to fit the rules in the help center, please edit the question.

7 Answers

One-person independent game. It was a multiplayer tank game with destructible terrain, and the destructible terrain and collision code proved somewhat flaky.

I ended up rigging up some basic dumb AIs (by "dumb", I mean "absolutely idiotic" - they would randomly choose "drive towards an enemy tank", "drive away from an enemy tank", and "drive in a random direction", while randomly firing the main weapon) and playing the game at maximum framerate while recording keypresses. I got about 10-15x realtime. The code was heavily asserted, so if anything went wrong, it would dump the entire keypress log to disk along with an error report and the initial random seed. I could then go and replay the keypress log to duplicate the state exactly, or just debug from the error report.

I left it running constantly for literally months. At the beginning it would rarely get an hour without crashing - I had to sit there and babysit it for a week, killing several obscure bugs per day. Eventually it got to the point where it was running for a week between failures, which translates to about 1500 player hours per crash.

It was invaluable and I heartily recommend it.

share|improve this answer
3  
+1 for the keylog output –  tenpn Jul 16 '10 at 8:09
    
Really slick! Yeah, the keylog is pure win. –  David McGraw Jul 17 '10 at 0:30
    
I'm confused: "rigging up some basic dumb AIs" and "recording keypresses" - who's keypresses? I thought You let Your AI play by itself without any humans? Did You actually let Your AI play the game by simulating keypresses rather than calling api functions? Now that would be slick! –  Dave O. Jul 17 '10 at 23:29
2  
@Dave Yeah I'll admit that's potentially confusing. The AIs were designed to provide all their output via simulated controllers. They took the game state as input, but didn't modify the game state in any fashion. This would probably have been a horrible idea with a mouse UI but the entire interface was done with gamepads, so it was a bit ugly but functional. I recorded the virtual keypresses of the AIs, and in addition, the same code recorded real keypresses when testing it with friends. –  ZorbaTHut Jul 18 '10 at 11:20
add comment

For an MMO I worked on (100ish developers, PC focused), we tried to add a huge variety of automated testing with varying success. Here's what worked:

  • Basic tests during our automated build process were a huge win. This included tasks such as creating a character, transferring maps, running some scriptable UI tests and looking for expected behavior. This caught a huge number of bugs before they actually got to the rest of the company.
  • On the server infrastructure end, we developed a bunch of different automated tests that simulated typical MMO server transactions. We could then play these back under a variety of circumstances to compare performance or ensure safety. Over time these tests became more and more accurate until it turned into a playback of live recorded data
  • We wrote a "fake player" that would randomly roam around the world, jump, kill things, and say random things in chat. This found a huge number of physics and infrastructure issues.

What didn't work:

  • We tried to add some very specific combat-oriented automated tests to the automated builder, but this basically never worked. It would work for about 3 days after being implemented, until a designer or artist changed something and the test would fail, throwing off the build failed alarms. 90% of the time it wasn't a real problem. These tests were too fragile, and actually testing specific gameplay on a specific map with specific powers can be unmaintainable
  • We tried implementing an automated performance test that would compare client performance (average FPS, etc) against recorded performance from a week ago. This was also fairly fragile as the demos we used for this tended to rot fairly often, and it was difficult to ascertain if a slowdown was caused by an actual loss in performance or some side effect of the test process.
share|improve this answer
add comment

Working on a 4x strategy game with 3d combat (think Homeworld meets Masters Of Orion) that unfortunately never saw the light of day as the company ran out of funding..

I always ensured that you could play the game without human players so we could leave the game running overnight.

We could turn off the 3d combat (simplified to a random result) and we left the AI strategy engine playing itself. This found numerous bugs and issues. Not only show stopper bugs but strategy bugs where the (eg) AI strategies would get deadlocked and spend 1000s of turns not doing "the right thing". These sort of bugs were difficult to spot just "playing the game".

share|improve this answer
    
Hm, I wouldn't have thought of this as automated testing -- but I guess you're right. I've been doing the same thing for a couple of years, just never thought of it that way. –  mmyers Jul 18 '10 at 5:20
add comment

On a first person shooter I worked on (Descent 3 -- linux/mac/windows, ~30 people on the team in 1999), the demo recording/playback capability turned out to be extremely useful. I made an option where you could playback the demo as fast as the game could render frames, and that became a great way to verify performance after a bunch of things changed.

It also exercised a lot of the code beyond the rendering system, so it was a nice sanity check. After making a bunch of changes I could just run the demo playback of 10 minutes of gameplay. Many times it would catch a bug in an area I wouldn't have thought to check myself.

share|improve this answer
add comment

We had a openworld shooter (x360,PS3,PC) that used a quick smoketest on the build server - it loaded the game, stepped through the front end, ran [the avatar] forwards, dumped a screenshot, and exited. If cctray detected the clean exit the build was a success.

We ran it for about the last year of the project, and with a team size of ~100 devs.

It was effective at catching showstopping bugs but it was easy to create a build that passed the smoketest but failed most "real" levels, or didn't work in multiplayer, or nobbled the AI, so it wasn't perfect. It was definitely worth doing.

I've heard since I left they've started running a larger range of smoketests, farmed out to multiple PCs. Apparently maintaining the smoketests is an issue, and there's a small team dedicated to just keeping the build servers and software maintained, so I can't say if that's been a success or not.

share|improve this answer
add comment

My experience with Automated Testing during the development of Crysis 2 is available here: http://yetanothergameprogrammingblog.blogspot.com/2010/06/aaa-automated-testing.html

Summary:

  • Automated testing improved deliverables stability, increasing productivity for both content creators' and engineers
  • Automated testing is an effective tool to improve code quality and reduce the chances of having to work overtime
  • The Game Industry as a whole is very reactionary in general, automated testing meets several irrational arguments against
  • Don't call it testing, call it something else, almost anything else (Look at Behavior-Driven-Development)
  • Be flexible, writing good tests is hard and require skills that are not widely available in the Game Industry
share|improve this answer
2  
Could you please consider updating your answer to provide a summary of your experiences? –  Alex Apr 13 '13 at 15:55
add comment

Game development is actually one of those cases where unit testing seems to make some sense to me, because the interactions between discrete systems are so common. Design-by-contract is of course a part of this, and should be planned on from day one of development, but I don't see why it couldn't be implemented later assuming the wherewithal to do so exists.

The hard part is, of course, integration testing. Lots of a game can be tested just by demo-looping it or something, but that stuff is conceptually fairly easy to debug--where I'd be more interested in spending my time is exposing bugs that will happen when a player does something, with the mindset that a bug the player never sees is obviously less important than a bug the player does.

Which is pretty difficult, obviously. Tactics that work on other applications (fuzzing, expected-pass/expected-fail, etc.) don't work so well here. In scriptable systems it seems like building a test set of scripts to simulate a player are the way to go (see JZig's answer). But testing specifically for stuff a player may encounter directly strikes me as the best place to focus your time for both human and automated testing purposes.

share|improve this answer
6  
But players never do what you would expect a sane person to do. That's why you don't see stuff like the elevator glitches in Call of Duty until after release. Because there's a thousand guys all doing stuff that the developers and the testers never thought to try. As soon as someone creates the perfect simulation of an obsessive-compulsive 16 year old gamer we'll have reached the game development singularity :) –  Casey Wagner Jul 18 '10 at 4:39
add comment

Not the answer you're looking for? Browse other questions tagged or ask your own question.