To expand on the data is king sentiment a bit (+1 to Tetrad!):
Investigate recording and playback:
- If your game is deterministic and frame-based you may only need to store an initial random seed and a tuple of
(key/button state, joystick/mouse coords, frame #) any time the input state changes. Playback is as simple as redirecting your input to this stream. (We've done this for many platform-jumping games in the past.)
- If your game has well-defined APIs or message systems for performing actions (a turn-based strategy game, a card game, a board game, or the like) you might be able to just harvest API calls or messages at a certain pinch point. (We did this for a card game for a handheld platform.)
- It's harder on some systems (less deterministic, threaded, or arbitrary timestep systems can be a pain) but it may be worth recording the data anyway; you can get "close enough" results for some uses.
A "replay" system based on these methods has a bunch of advantages:
- use it to reproduce crashes in a debug or otherwise instrumented build or environment;
- load replays under a profiling build and get performance or resource usage data;
- wire it into the game to provide "instant replay" functionality, maybe with a different camera or time step;
- set up an "attract mode" demoing gameplay if the user is sticking around doing nothing on a menu;
- put it on your build system as a smoke test: if I can play through this replay without crashing, it's more likely a good build;
- watch examples of people playing to see what they did and didn't do.
Wire in random input rather than a recorded stream, and you've got a great monkey test that you can leave soaking overnight or whenever your development machines are idle.
Next, do some event recording. For a hypothetical FPS, start with something like "time T: X killed Y at point Z with weapon W": put it in a log.
Once you have some data being collected, figure out how to automate collection. It doesn't have to be elegant during development:
- connect to an SQL server and insert rows,
- fire and forget UDP packets at some simple syslog-ish server,
- e-mail the log the next time the game boots,
- just wrap the executable in a shell script or batch file that renames and copies a .log file to a common shared drive,
- (later, for production builds) use Windows Error Reporting or a similar service to collect crash data...
Doesn't really matter, as long as you can gather data.
Now extend it: gather crash dumps, stack traces, and input or event recordings. Add more events, and more data sources:
- sample the player position or hand every 10 seconds, plot it on a map -- "hey, no-one's using this corner that I spent a week modeling, time to put a powerup there"
getFreeMemoryBytes() every half-minute
- take a photo or a video of what the user is doing via a webcam (great for automated usability testing -- only with user permission and understanding, of course)
- grab system info (again, with user permission)
The "plot it on a map" thing can get really awesome after a while: envision an aerial view of an RTS or FPS map. Put a slider on it, representing time since start of game. Select an event type ("got gold", "killed someone", whatever). Select a data set: maybe one game, maybe 500 games over the last few months. Start pulling the slider to the right and watch the activity pop onto the map.
And if you can't find good libs to help you with this stuff (there are quite a few here and there, though!), consider rolling your own: it's a good learning experience, and it doesn't need to be particularly elegant to be useful.
Get the data, you'll figure out what to do with it. =)