6
\$\begingroup\$

I have a multiplayer server made by a small team of developers and we have bugs that crop up on our live server occasionally that we simply can't reproduce while playing with our small team on a live server.

It clearly isn't feasible to attach a debugger and halt the world on a live shard... so what are some effective strategies in debugging multiplayer games while they are still running?

Attaching a debugger is fine on a developer server, but perhaps not always feasible to reproduce the bug or always effective.

\$\endgroup\$
1
  • \$\begingroup\$ You can use crash/core-dumps \$\endgroup\$ Commented Jan 3, 2013 at 23:48

2 Answers 2

10
\$\begingroup\$

Configure your servers to capture and save crash dumps (if you run on Windows, MiniDumpWriteDump and SEH is a good place to start) so that you can diagnose hard crashes offline. You'll probably want to make sure your release builds generate and archive a PDB or other symbol- and debug-help related data internally as well as provide a mechanism to allow you to correlate the crashed build ID and the proper PDB (et cetera) for in-house examination.

You may want to familiarize yourself with debuggers other than Visual Studio -- later versions of VS are much better at just "doing the right thing" with a minidump but it's also worth looking at WinDbg, which is a standalone debugger that's more expressive (but much harder to learn to use). It's saved me a lot of trouble on occasion.

For code in C#, for example, minidumps plus the SOS extension for WinDbg make for a very useful postmortem debugging environment for managed code.

Additionally, the process that collects the crash dumps should probably report metrics back to you somehow so you can monitor how frequently you are getting live crashes.

As for handling non-crash bugs, build logging and metrics gathering tools. This is harder to do after the fact, but possible (it's much easier if you integrate the construction of these metrics into the construction of the software itself, but the ship has probably sailed on that option by now).

You want to allow for the logging of just about anything. In an ideal world, you want to support the ability to add logging after the fact without having to reboot the entire (player-facing) server. Whether or not this is possible depends largely on how your server infrastructure handles failover -- if you can bring a new build of the server executable (with new logging) up and transparently migrate users over there before beginning your analysis, that'd be best. Of course, that's not a small task to accomplish either.

But at the very least you'll want your logging to be toggle-able -- you don't want to log everything, everywhere, for every user, all the time. You should gate the ability to enable the log to a server-side check of a user's rights (so only developer or GM users can toggle it) and only communicate that log data through a private, back-end channel that your developers can access from the privileged side of the network.

On top of the logging you can build your own custom live debugging UI, if you log in a structured enough form. This is essentially a debugger you can connect (again, only through the internal side of the network) that parses the log data (when enabled) and displays it in a more human-readable form, but does not halt execution of the server process itself. That way you disrupt other players minimally.

Obviously you can tailor the specificity of this to your needs and your actual problem. I'd recommend that approach actually -- build a log/debug view that's very specific to the actual problem you are having and generalize it after the fact. That way you focus on the real issue first and don't get lost trying to build an overly complex generalized system with features you may not yet need.

\$\endgroup\$
0
\$\begingroup\$

Starting to log crash dumps are a good start, but it might not tell you the whole story. Your users have a diversity of hardware/network connections and may use the game in ways you may not have imagined. The crash dumps may tell you WHAT the error was, but not what the conditions on the clients or even your server were at the time the crash dump occur.

To help you establish a full picture, you'll want to make use of logging and log not only crash dumps from your game server software but also server environment variables and things on your client related to how it interacts with your server.

From my experience running games, you don't want to get involved in creating your own logging tools. Log data can swell up into the giga or terabyte level, and between managing that volume of data AND building a tools, you'll end up using a lot of your server guys simply to maintain logging. It's a lot quicker to just use a 3rd party logging service like Loggly to get up and running.

Regardless of the path you pick to collect, here are the logs you’ll want to consider collecting:

Game server application logs: You can log most any information from your application to file or SYSLOG socket. So within your application code, log any crash or exception information. Also think about logging your gameplay mechanics for things that would indicate when they're failing so that you get a list of events that led to the error.

General server logs: Most everything from the server, system logs, web server logs, database logs, etc. This data can help you spot any abnormalities in server performance which might indicate why a bug is happening.

Client logs: A lot of online game bugs stem from high latency caused by a slow client-side network or from bugs in the client sending incorrect data or commands to the server. You want to log anything within the client that has to do with communication with the server or manipulation of user's data to catch these client-driven bugs.

As far as how you'd collect the log data itself Try to avoid storing logs directly on live servers, logs blow up to huge volumes, and you really don't want to be managing log retention & purge issues on live servers. For anything you want to log you can choose to write it to file and then have a SYSLOG agent or send it directly to a SYSLOG socket and have it sent out to a provider like Loggly directly. For logging anything on the client, you'd simply store any data you'd want to log in JSON format and send it direct from the client using HTTP methods.

It sounds like you don't already have logging implemented, so you'll need to implement it into your server software and take it live. It would be elegant to somehow push this update while your game is live such it doesn't interrupt player experience, but if you don't have a way of doing this, I wouldn't spend a lot of effort on creating it just yet; one problem at a time. Just take down the servers at a time you know there are a low volume of players and patch logging-enabled updates then.

\$\endgroup\$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .