37
\$\begingroup\$

Do gamers developers use to write unit and integration tests? How common is it among puzzle developers? And among developers of MMORPGs and FPSes?

(I have no background in game development neither am I cogitating about working with it - it is just a doubt that occurred to me. So, no need of trying to convince me to write them, even because I already like to write automated tests.)

\$\endgroup\$
4
  • 3
    \$\begingroup\$ How common are automated questions on Stack Exchange? \$\endgroup\$
    – House
    Commented Dec 22, 2011 at 6:03
  • 2
    \$\begingroup\$ possible duplicate of Automated testing of games \$\endgroup\$
    – bummzack
    Commented Dec 22, 2011 at 8:26
  • \$\begingroup\$ Just because they're not common in the games industry doesn't mean you shouldn't strive to continue writing them. What problem are you trying to solve with this question anyway? \$\endgroup\$
    – Tetrad
    Commented Dec 23, 2011 at 3:59
  • \$\begingroup\$ @Tetrad Just read the question. The second paragraph explains all. \$\endgroup\$
    – brandizzi
    Commented Dec 23, 2011 at 11:34

7 Answers 7

40
\$\begingroup\$

In general, unit and integration testing of games isn't that common. This is mostly because the rendering aspect of games is usually closely tied to the rest of the game mechanics that it can be very hard to actually write unit tests that work.

That said, unit testing can happen in game development, and if the code is set up for it, it can be of great benefit. However, it can be much more common to write automated tests for games, usually in the form of an AI program that can effectively play the game at a higher speed than a normal player. There's some excellent stories of developers doing just that in this question about automated testing. This sort of automated testing is potentially better, since unit testing might not catch a bug in the rendering engine, but an automated test is more likely to expose such a problem.

In the end, though, this all depends on the studio. Some studios will some sort of automated testing, while others might just hire 20 high school kids over the summer to play their game for hours on end.

\$\endgroup\$
4
  • 18
    \$\begingroup\$ +1 just because we all wish we were one of these kids. :( \$\endgroup\$
    – Bobby
    Commented Dec 22, 2011 at 9:33
  • 13
    \$\begingroup\$ @Bobby Speak for yourself. Being forced to play buggy and unfinished games over and over is a horrific thought, even if you don't end up being assigned to test something like the latest Barney the Dinosaur game. \$\endgroup\$ Commented Dec 22, 2011 at 14:45
  • 10
    \$\begingroup\$ @Bobby, I was QA for around 3-4 years, it's a great job if you like breaking software and working in that industry, but don't do it because 'you love playing games all day' :) \$\endgroup\$ Commented Dec 22, 2011 at 16:49
  • 10
    \$\begingroup\$ I was in QA for about six months. While walking to my car at the end of my second day on the job, I vividly remember thinking to myself, "I can see myself eventually coming to hate this job." And at the end of my third day, "I really hate this job." Good QA testers who can cope with the demands of the job are worth their weight in gold, and it's a crime that they aren't valued and compensated more highly than they are. \$\endgroup\$ Commented Dec 23, 2011 at 0:41
19
\$\begingroup\$

In my experience, it's not very common.

Mostly unit testing doesn't occur because most game developers come from a time and culture before things like that were widespread, and therefore it's hard to make the argument now that such methods are necessary. This has become even more true in recent years with the expectation that the user is able to patch his own software after release.

Partly it's because the dominant language in the game development industry is C++ and that makes unit testing a little more cumbersome than other languages. Unit testing frameworks exist but they are not as easy to use as similar systems in more modern languages that can use reflection and similar tricks to speed up the detection of test cases.

Also, it's because games don't generally lend themselves to unit testing - much of the logic depends on semi-deterministic sources (eg. graphics hardware, input timings, the frame rate), much of the output is hard to measure (eg. on-screen graphics, sound effects) and some is almost meaningless outside of the full game context making (eg. complex reactive AI, physics simulations). There are exceptions - many, if you work hard to make the code that way - but on the whole testing is more expensive in games than in most other types of software and so the cost/benefit ratio is more dubious.

As for integration testing, I've never heard of the term being explicitly used in game development but many developers do conduct automated tests of the whole system where possible. At a guess I'd say maybe 1 in 3 pro developers do this, as it's not always easy to set up, and because the benefits are reduced by the fact that pretty much every medium to large developer (or their publisher) has a QA department that will manually perform similar tests. QA are typically paid much less than developers so it may be considered economical to leave testing to them, rather than to invest extra code time on it. (Controversial.)

\$\endgroup\$
1
  • 5
    \$\begingroup\$ Great point about the output being hard to measure. It's easy to automatically test that a class spits out, say, syntactically correct JSON, but the only way to ensure that your ragdolls don't spin out of control when shot is to actually play the game. The worst part is that this makes it really really difficult to notice side effects (ie. you fix a bug, but now some other remote part of the game behaves differently.) \$\endgroup\$
    – jhocking
    Commented Dec 22, 2011 at 15:33
9
\$\begingroup\$

In general automating UI testing (even in regular programs) is harder than regular automation. So even though you can write unit tests for your games, testing the actual game is harder. Most companies use human testers that run the game time and again.

For example Here is an article explaining how they do it in a small Game Studio (2 devs). From what I read it seems like their validation is not very detailed (automation starts the game and logs any errors/asserts).

However some companies are very successful with semi automation (Such as Microsoft Test Studios). That is, Developers or SDETs build tools that make testing the game much easier. There have been Gamefest talks where they discuss how they tested Crackdown for example or Fable. For example, they still use testers that verify that every object is where it's supposed to be but they use a tool that takes a snapshot of that location so that all that the human does is visually verify it is there without having to play the game.

Here is a good talk on what kind of tools they build/use to test the games. It's called "Test Lead Gems: Getting Out in Front of the Increasing Complexity of Our Games"

\$\endgroup\$
8
\$\begingroup\$

I don't see how writing a game is different to any other piece of software as far as testing goes. Each component of the software, whether it be triggering a timed event, sending commands to an in-game character, or pressing menu buttons, should be tested for proper execution.

Testing whether the game is possible to complete is a different matter and doesn't fall under unit or integration testing so much.

\$\endgroup\$
4
\$\begingroup\$

I participated in a roundtable discussion about automated testing at GDC 2011. IIRC, there were about 60 people in the room. At one point the moderator took a survey of unit test coverage. There was one person who claimed greater than 90% code coverage. Everyone else laughed at the thought of ever reaching 1% coverage. If that group is a fair representation of the industry as a whole, I'd say automated testing generally doesn't happen much, if at all.

The other answers here give good reasons as to why. I just thought it would be useful to have a first-hand account.

\$\endgroup\$
2
  • \$\begingroup\$ I'm surprised the figure is so low (though I wouldn't expect more than a third of gamedevs to use such tests, as I said in my answer.) To add some personal anecdotal evidence, the server software I'm working on has over 70% unit test coverage. I could probably get it to 85% with a bit of hard work, but the last 15% would involve various dependency injection contortions that I'm not willing to make. By comparison, the client software is almost impossible to unit test so we focus on manual testing. \$\endgroup\$
    – Kylotan
    Commented Dec 22, 2011 at 22:58
  • \$\begingroup\$ On a Lua project, thanks to easiness of stubbing and mocking, I managed to keep 100% coverage during development. However, I noticed many tests were uninteresting (such as testing exact placement of UI, or anything that should be data-driven but happened to be done in code really). To keep things cleaner, I split the code between "engine" (reusable) and game-specific, and only made sure to cover all the engine code, while coverage fluctuates for game code (I still test low-level classes as it's easy to do, and custom physics as it's easy to mess up; but not high-level UI/rendering anymore). \$\endgroup\$
    – hsandt
    Commented Nov 14, 2019 at 0:47
4
\$\begingroup\$

I would wager that MMO and multiplayer server code, however, is a bit more often tested.

At the very least, automated regression tests have been common. I've seen these implemented as mass sanity-checks during server start-up, to e.g. make sure that a new "cloud" server was configured correctly before it begins accepting players; a fairly good regression suite built up over 3-4 years, in that case, ran in about 4 seconds, while bringing up a virtual host (from a blank OS image) took almost 10 minutes, so it was well worth the time. We ran the same tests on a "tinderbox" (continual build system) on our Subversion repository to check for some annoying, fairly common errors that liked to creep back in. In particular, the multi-server functionality had a nasty habit of trying to create duplicates of objects as they were passed around: the object instantiation, caching, and network-passing code was close to 100% covered; we kept thinking we'd thought of everything that could be tested, and then discover some “fun,” new edge case.

At several MMO's that I've worked on, we also would develop "stub clients" to perform initial unit-testing, and usually provided "operator" commands to do ad-hoc unit testing of new features. This let us execute server code before the client was ready to take advantage of it, and exercise "impossible" situations (e.g. teleporting a player inside of a wall) to ensure that the error recovery handlers would work well. Bringing a new feature online on the server might sometimes take many days less than the client support for it; conversely, we'd sometimes have to create a "dummy" server method for the client, returning fake-but-well-formed data, if they got ahead of us.

However, MMO development in general is subject to a lot more of these kinds of problems, which might reflect the environment. When I was working on embedded game systems, “testing” was practically unheard-of for anything except some reusable widget code (e.g. text editors).

\$\endgroup\$
3
\$\begingroup\$

Another reason why automated testing is not that common in Game Development that may be considered is that for most games there are plenty of Beta testing volunteers who preceive Game Beta participation as an "edge" when the game is released. Automated testing has of course grown out of quality requirements, but also out of budget restrictions. Therefore, since in gaming there are plenty of experienced testers who are prepared to test for free, perhaps this is another reason automated tests are not that widespread.

\$\endgroup\$

Not the answer you're looking for? Browse other questions tagged .