Take the 2-minute tour ×
Game Development Stack Exchange is a question and answer site for professional and independent game developers. It's 100% free, no registration required.

I have a multiplayer server made by a small team of developers and we have bugs that crop up on our live server occasionally that we simply can't reproduce while playing with our small team on a live server.

It clearly isn't feasible to attach a debugger and halt the world on a live shard... so what are some effective strategies in debugging multiplayer games while they are still running?

Attaching a debugger is fine on a developer server, but perhaps not always feasible to reproduce the bug or always effective.

share|improve this question
    
You can use crash/core-dumps –  Maik Semder Jan 3 '13 at 23:48

1 Answer 1

up vote 7 down vote accepted

Configure your servers to capture and save crash dumps (if you run on Windows, MiniDumpWriteDump and SEH is a good place to start) so that you can diagnose hard crashes offline. You'll probably want to make sure your release builds generate and archive a PDB or other symbol- and debug-help related data internally as well as provide a mechanism to allow you to correlate the crashed build ID and the proper PDB (et cetera) for in-house examination.

You may want to familiarize yourself with debuggers other than Visual Studio -- later versions of VS are much better at just "doing the right thing" with a minidump but it's also worth looking at WinDbg, which is a standalone debugger that's more expressive (but much harder to learn to use). It's saved me a lot of trouble on occasion.

For code in C#, for example, minidumps plus the SOS extension for WinDbg make for a very useful postmortem debugging environment for managed code.

Additionally, the process that collects the crash dumps should probably report metrics back to you somehow so you can monitor how frequently you are getting live crashes.

As for handling non-crash bugs, build logging and metrics gathering tools. This is harder to do after the fact, but possible (it's much easier if you integrate the construction of these metrics into the construction of the software itself, but the ship has probably sailed on that option by now).

You want to allow for the logging of just about anything. In an ideal world, you want to support the ability to add logging after the fact without having to reboot the entire (player-facing) server. Whether or not this is possible depends largely on how your server infrastructure handles failover -- if you can bring a new build of the server executable (with new logging) up and transparently migrate users over there before beginning your analysis, that'd be best. Of course, that's not a small task to accomplish either.

But at the very least you'll want your logging to be toggle-able -- you don't want to log everything, everywhere, for every user, all the time. You should gate the ability to enable the log to a server-side check of a user's rights (so only developer or GM users can toggle it) and only communicate that log data through a private, back-end channel that your developers can access from the privileged side of the network.

On top of the logging you can build your own custom live debugging UI, if you log in a structured enough form. This is essentially a debugger you can connect (again, only through the internal side of the network) that parses the log data (when enabled) and displays it in a more human-readable form, but does not halt execution of the server process itself. That way you disrupt other players minimally.

Obviously you can tailor the specificity of this to your needs and your actual problem. I'd recommend that approach actually -- build a log/debug view that's very specific to the actual problem you are having and generalize it after the fact. That way you focus on the real issue first and don't get lost trying to build an overly complex generalized system with features you may not yet need.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.