Configure your servers to capture and save crash dumps (if you run on Windows, MiniDumpWriteDump and SEH is a good place to start) so that you can diagnose hard crashes offline. You'll probably want to make sure your release builds generate and archive a PDB or other symbol- and debug-help related data internally as well as provide a mechanism to allow you to correlate the crashed build ID and the proper PDB (et cetera) for in-house examination.

You may want to familiarize yourself with debuggers other than Visual Studio -- later versions of VS are much better at just "doing the right thing" with a minidump but it's also worth looking at WinDbg, which is a standalone debugger that's more expressive (but much harder to learn to use). It's saved me a lot of trouble on occasion.

Additionally, the process that collects the crash dumps should probably report metrics back to you somehow so you can monitor how frequently you are getting live crashes.

As for handling non-crash bugs, build logging and metrics gathering tools. This is harder to do after the fact, but possible (it's much easier if you integrate the construction of these metrics into the construction of the software itself, but the ship has probably sailed on that option by now).

You want to allow for the logging of just about anything. In an ideal world, you want to support the ability to add logging after the fact without having to reboot the entire (player-facing) server. Whether or not this is possible depends largely on how your server infrastructure handles failover -- if you can bring a new build of the server executable (with new logging) up and transparently migrate users over there before beginning your analysis, that'd be best. Of course, that's not a small task to accomplish either.

But at the very least you'll want your logging to be toggle-able -- you don't want to log everything, everywhere, for every user, all the time. You should gate the ability to enable the log to a server-side check of a user's rights (so only developer or GM users can toggle it) and only communicate that log data through a private, back-end channel that your developers can access from the privileged side of the network.

On top of the logging you can build your own custom live debugging UI, if you log in a structured enough form. This is essentially a debugger you can connect (again, only through the internal side of the network) that parses the log data (when enabled) and displays it in a more human-readable form, but does not halt execution of the server process itself. That way you disrupt other players minimally.

Obviously you can tailor the specificity of this to your needs and your actual problem. I'd recommend that approach actually -- build a log/debug view that's very specific to the actual problem you are having and generalize it after the fact. That way you focus on the real issue first and don't get lost trying to build an overly complex generalized system with features you may not yet need.

Configure your servers to capture and save crash dumps (if you run on Windows, MiniDumpWriteDump and SEH is a good place to start) so that you can diagnose hard crashes offline. You'll probably want to make sure your release builds generate and archive a PDB or other symbol- and debug-help related data internally as well as provide a mechanism to allow you to correlate the crashed build ID and the proper PDB (et cetera) for in-house examination.

You may want to familiarize yourself with debuggers other than Visual Studio -- later versions of VS are much better at just "doing the right thing" with a minidump but it's also worth looking at WinDbg, which is a standalone debugger that's more expressive (but much harder to learn to use). It's saved me a lot of trouble on occasion.

For code in C#, for example, minidumps plus the SOS extension for WinDbg make for a very useful postmortem debugging environment for managed code.

Additionally, the process that collects the crash dumps should probably report metrics back to you somehow so you can monitor how frequently you are getting live crashes.

As for handling non-crash bugs, build logging and metrics gathering tools. This is harder to do after the fact, but possible (it's much easier if you integrate the construction of these metrics into the construction of the software itself, but the ship has probably sailed on that option by now).

You want to allow for the logging of just about anything. In an ideal world, you want to support the ability to add logging after the fact without having to reboot the entire (player-facing) server. Whether or not this is possible depends largely on how your server infrastructure handles failover -- if you can bring a new build of the server executable (with new logging) up and transparently migrate users over there before beginning your analysis, that'd be best. Of course, that's not a small task to accomplish either.

But at the very least you'll want your logging to be toggle-able -- you don't want to log everything, everywhere, for every user, all the time. You should gate the ability to enable the log to a server-side check of a user's rights (so only developer or GM users can toggle it) and only communicate that log data through a private, back-end channel that your developers can access from the privileged side of the network.

On top of the logging you can build your own custom live debugging UI, if you log in a structured enough form. This is essentially a debugger you can connect (again, only through the internal side of the network) that parses the log data (when enabled) and displays it in a more human-readable form, but does not halt execution of the server process itself. That way you disrupt other players minimally.

Obviously you can tailor the specificity of this to your needs and your actual problem. I'd recommend that approach actually -- build a log/debug view that's very specific to the actual problem you are having and generalize it after the fact. That way you focus on the real issue first and don't get lost trying to build an overly complex generalized system with features you may not yet need.