How do you get useful data from playtesters? [closed]

Question

There are a few types of feedback you can get from playtesters, and I wonder how to best gather data for each of them...

1. Crash Reports. When my C++ game crashes while someone is playing it, how do I best make sure that I know about it and even better... what caused it? Even getting something as simple as the file and line number which caused a crash would be incredibly helpful.

2. Design Feedback. When a play tester is playing the game, how can I figure out if they are having fun, why they are having fun, why they aren't having fun, and what we should spend time adjusting?

Answer 1

I'm assuming you're talking about on-site playtesters and not internet beta testers.

Rule #1: Don't help them. Frustration should be the top thing you should be checking for. The ideal situation would be a two way mirror with your team on one side and the playtester on another with one video camera on their face and another on the screen. Obviously this isn't feasible for most people, so do the best you can. Just having your designers sit and watch where people get stuck is very useful information. You're not going to be standing over their shoulder when they take the game home, so you giving advice on how to pass certain sections isn't going to give you the information you need. Edit: another way of putting it is this: Don't think they're "Playing it wrong"

Rule #2: Don't give them what they want. After a playtest session you have some kind of questionnaire that they fill out. The specific suggestions they have are usually not wise to take at face value. Usually there's some root cause that is triggering most responses and they just don't know how to express it. If you can figure that out, you'll be better off doing it. Although at the moment I'm having trouble coming up with specific examples.

Rule #3: Data is king. If you can (and this is really another wishlist item, honestly), track everything you can. Track where players die. Track where they run out of ammo from a specific gun. Track what pickups they miss. Track what upgrades they buy. Track what enemies do damage to them. Obviously these are FPS-specific examples, but I'm sure there are domain specific ones for whatever game you're doing. If everybody's doing something or not doing something, those are usually areas that you should spend a little more time looking at.

Basically, you don't care what player's think. You care about getting raw numbers for what players do. You need virgin eyes to see your game and tell you what makes them frustrated and what they're being led to do.

---

For crashes, investigate minidumps. They're not perfect, but are a very useful tool to figure out where crashes are.

Also consider a built-in bug reporting tool. Something where the user can take a screenshot , add a description, and email it to someone automatically from inside the game. Ideally with a snapshot of the world (i.e. quicksave or some kind of memory dump) if your game supports it.

Answer 2

To expand on the data is king sentiment a bit (+1 to Tetrad!):

Investigate recording and playback:

• If your game is deterministic and frame-based you may only need to store an initial random seed and a tuple of (key/button state, joystick/mouse coords, frame #) any time the input state changes. Playback is as simple as redirecting your input to this stream. (We've done this for many platform-jumping games in the past.)
• If your game has well-defined APIs or message systems for performing actions (a turn-based strategy game, a card game, a board game, or the like) you might be able to just harvest API calls or messages at a certain pinch point. (We did this for a card game for a handheld platform.)
• It's harder on some systems (less deterministic, threaded, or arbitrary timestep systems can be a pain) but it may be worth recording the data anyway; you can get "close enough" results for some uses.

A "replay" system based on these methods has a bunch of advantages:

• use it to reproduce crashes in a debug or otherwise instrumented build or environment;
• load replays under a profiling build and get performance or resource usage data;
• wire it into the game to provide "instant replay" functionality, maybe with a different camera or time step;
• set up an "attract mode" demoing gameplay if the user is sticking around doing nothing on a menu;
• put it on your build system as a smoke test: if I can play through this replay without crashing, it's more likely a good build;
• watch examples of people playing to see what they did and didn't do.

Wire in random input rather than a recorded stream, and you've got a great monkey test that you can leave soaking overnight or whenever your development machines are idle.

Next, do some event recording. For a hypothetical FPS, start with something like "time T: X killed Y at point Z with weapon W": put it in a log.

Once you have some data being collected, figure out how to automate collection. It doesn't have to be elegant during development:

• connect to an SQL server and insert rows,
• fire and forget UDP packets at some simple syslog-ish server,
• e-mail the log the next time the game boots,
• just wrap the executable in a shell script or batch file that renames and copies a .log file to a common shared drive,
• (later, for production builds) use Windows Error Reporting or a similar service to collect crash data...

Doesn't really matter, as long as you can gather data.

Now extend it: gather crash dumps, stack traces, and input or event recordings. Add more events, and more data sources:

• sample the player position or hand every 10 seconds, plot it on a map -- "hey, no-one's using this corner that I spent a week modeling, time to put a powerup there"
• getFreeMemoryBytes() every half-minute
• getFPS() periodically
• take a photo or a video of what the user is doing via a webcam (great for automated usability testing -- only with user permission and understanding, of course)
• grab system info (again, with user permission)

The "plot it on a map" thing can get really awesome after a while: envision an aerial view of an RTS or FPS map. Put a slider on it, representing time since start of game. Select an event type ("got gold", "killed someone", whatever). Select a data set: maybe one game, maybe 500 games over the last few months. Start pulling the slider to the right and watch the activity pop onto the map.

And if you can't find good libs to help you with this stuff (there are quite a few here and there, though!), consider rolling your own: it's a good learning experience, and it doesn't need to be particularly elegant to be useful.

Get the data, you'll figure out what to do with it. =)

Answer 3

Ofcourse this depends a lot on... a) what kind of testing you want to have done, and b) what kind of game you are testing, and c) what kind of testers and infrastructure you have available...

It also differs a lot if you are testing for a) functionality, b) balancing c) game-design

But in general you might want to consider these aspects...

*a) Choose the right person for the job Sounds too simple to mention, but i've seen it many times and just have seen it live again. As always, there are significant differences between people in regard to how good they are at different jobs. Some people who are willing or maybe eager to do testing might not be playing thorough enough to find unusual (or even simple) bugs. Some are not good at describing the bugs. Some are better at testing balancing issues, some are more attentive to visual weaknesses, some are more creative in playing the game in unusual ways and finde hidden/rare bugs, some are more attentive to technical or visual quality, some are better in understanding aspects of the game mechanic and may be even able to suggest meaningful changes (if you want your testers to do that ;).

*b) Use an Issue-Tracker / Bug-Tracker Software These tools can not only help in organizing your issues but also in increasing the quality of the output of your testers by giving them a frame to work within and by learning from feedback they get from developers about their issues. It helps to improve the quality of the output of your testers a lot faster than if you work without it. (It also helps a lot with remote testers) Typical software used by game studios is i.e. Mantis, JIRA, (and ofcourse lots of others..) See Wikipedia for a general list and also this post on SO.

c) Add ingame testing tools Typical are Shortcuts to test specific levels or sections of the game. Displaying extra information during the game to the testers so that they can add this to bugreports. This could be the position in a level, the number of active objects in a scene, the amount of texture-ram or palettes currently in use, anything helpful to the developers.

d) Combine experienced testers with fresh blood Always a good thing to have testers who are very experienced with your game and have learned what the typical problems are and how to (re)-test them. At the same time you want new "virgin" players once in a while, especially for balancing.

e) Have a Test Manager Someone who coordinates the process and tailors it to the game at hand, the current priorities and the available testers and testing environment.

f) Have a Testplan Document This would be worth an extra post.

Answer 4

As Tetrad mentioned, get as much objective data as you can. Putting in hooks to store certain events and dump those all to a .csv isn't very hard. And once you've got it in Excel, you can study, graph and plot until the cows come home.

Also, have specific questions you're looking to answer. Scientists don't just sit down, fire up some experiments and "do science." They have specific, measurable questions they want data about. You'll often get the most out of playtesting if you take the same approach. Trying to figure if your game is "good" is very hard to quantify. But figuring out if the simple tutorial mission is only taking the 5 minutes you expect, or if the testers are struggling to solve a certain puzzle, can actually be evaluated.

Sometimes, the most effective way to test is iteratively in short bursts. Have a couple of testers go for an hour or so in the morning, make some changes to the issues they identify and go again with new testers the following morning. Obviously you have to be looking at a feature small enough to improve in an afternoon. But for a problem that's been particularly stubborn, this method can be very successful.

Answer 5

Definitely Agree with Tetrad's Rule #1. Do not help them. I would say a caveat is to explain to them that you will not be helping them, and if they need help to please ask. This way the player doesn't end up being frustrated.

The Questionnaire should ask open ended questions, instead of ones that are simple yes/no, depending on the age and number of testers these can be forms they fill out, or you can ask the questions. Its also important to ask questions about their history and their familiarity with the genre of game you are having them test; this will add context to their specific answers about your game.

Fortunately when my game crashes it gives me a dump of information regarding the error, so I can take a picture of that and make a note of what the player was doing. Normally I test younger age groups so when we get a crash I have to remember to explain to them that they are doing a great job -- they can get upset after breaking the game. Playtesters have really proven to be great at finding obscure bugs that the dev team would never come across playing the game the "right" way.

Answer 6

It's possible to write code that catches crashes and logs the call stack. That can help a lot with bug reports. Having a useful log file generated helps too. You can prompt the user to send these files the next time they play, or have a standalone tool that runs after the game is closed or crashes if you like.

Answer 7

For crash reports, you should be relying on paid QA staff, not playtesters. QA is someone you hire specifically for their ability to find bugs and report them in a meaningful way, and a good tester is worth their weight in gold (and costs only a tenth of their weight in gold, compared to programmers!).

If you're worried about "oh, what if they accidentally discover a crash, we wouldn't want to miss out just because they're not testing for that"... this is what logging is for. A sufficiently good logging system should be recording enough elements of play to be able to reproduce a crash exactly.

For design feedback, there's no substitute for having your game designers actually watch them play (or use a video recorder if you must, etc.). Don't rely on playtester memory or opinion, both of which are notoriously poor. But if you're watching them play in real time, it is patently obvious just by looking at their face and body posture whether they're engaged, bored, or frustrated.