Is networked gameplay inherently hard to debug and create? [closed]

Question

I have worked with XNA before and made a simple networked game. I have also worked with Unity3D as well. One thing I notice on many game engine web pages is that they talk about how the engine is great for networking and makes it easier. I then see on these pages links to articles by Valve and such about network predictions and lag which I have seen a bit of with my XNA networked game.

My question is, is networking a game inherently hard to debug and similar to design no matter what engine/framework you use? For example is making a 4 player game with XNA or unity or UDK the same thing in terms of networking code or do some engines relieve this burden.

Answer 1

This is largely an opinion post, as "difficulty" is subjective, some people find things difficult that others find easy, and vice-versa. And honestly networking of games is such a huge topic I'd probably have to write a book about it to fully answer your question.

The difficulty of networking a game depends largely upon the game type. The most difficult would probably be real-time MMOs (massively multiplayer online game), and then FPSs (first-person shooter game), then RTSs (real-time strategy game). Turn-based games like Chess, or other strategy games are usually state-based and much easier to network.

Debugging difficulty depends on a lot of factors:

• Sending and receiving information during live gameplay: If a game is always updating the game world based on the new information, that means that when you want to debug you usually need either conditional breakpoints with some very specific conditions that you're expecting, or you'll need some kind of real-time output that you can read through to see when something occurs. You can also create large log files, outputting all the relevant information, and sift through those log files looking for the data you need. For a turn-based game, information is usually only sent when an action occurs. This means the game is usually sitting in a waiting state, waiting for input from a player. I've found this makes it very easy to watch the chain of actions that occur, and the data going back and forth during each action, to see where something goes wrong.
• Number of players: The more players that are interacting, the more information is being generated, and the more places are sending and receiving that data, and changing that data, etc. More players in a game at one time will almost always add to the difficulty of networking and debugging that networking.
• Precision: Depending on the type of game and the quality level you're shooting for, the precision of your network prediction may have to be high, or not. For a turn-based game this is usually not important, there is a relatively large amount of time between events (hundreds to thousands of milliseconds), and the number of possibilities in a single turn in a turn-based game is relatively small, so prediction can usually be 100% accurate. With something like an FPS game there will almost always be too much going on, in such small amounts of time, in an unpredictable manor, that you will need to make educated guesses on what position something will be in given a handful of variables, most important of which is usually time. Making a basic guesses during prediction will give you some illusion of precision, but can often be far from precise, and if you want something that is more precise most of the time then you'll need to put a lot more work into your prediction methods. Debugging this often means seeing what the server actually did, and what the client did, and looking for differences. Did the client have current information, if not when did it last receiving information, what did it do with the information, etc. Generally I would log a new event that came down from the server, and then log changes to that data, over a period of time, to watch what happened to the data, and see if it's what is expected, and if not I can usually see where the changes start to go wrong.
• Bandwidth: This goes hand-in-hand with the last point, which was 'Precision'. If you want very precise movement you often need to send more information per packet and/or more packets over time. Better prediction methods can often result in more accuracy without the need for as much information, but sometimes at the expense of more CPU usage on the client or server.
• Latency: If you're making an MMO, FPS, or some other game with networking occurring during gameplay then you may want to have quick reactions and adjustments made based on actions in the game. For example, if a player fires a weapon you're going to want all of the clients to see that as soon as possible. Unfortunately you usually won't get to pick the latency that clients have so your prediction algorithms will need to compensate for different latency between different clients. This becomes more difficult when there are wide ranges of latency between clients on the server. For example if player #1 has 100ms of latency, and they think they just shot player #2 at 30.1 seconds into game, and player #2 has 300ms of latency, and they think they just shot player #1 at 30.3 seconds into the game, which does the server choose? You received player 1's packet 0.2 seconds before player 2, but based on their latency times they probably both sent their packets at the same real time. On top of that problem, you may now realize that after receiving the packet from player 1 at 30.1 seconds into the game, if you had immediately responded to that packet by giving player 1 the reward for the kill, then what do you do when you get the packet from player 2? Just deny them the kill even though they shot player 1 at the same time? Luckily if you're making a turn-based game latency will be of almost no consequence other than to slow the game down ever so slightly by adding latency to the time between events. Debugging latency issues, luckily, is not as difficult of a problem as implementing the handling for latency in the first place. Usually you will have a queue of events that the client has said they have done, and timestamps for each event. As you're going through the queues you should be able to easily visualize the timeline of what has occurred from each client.
• Combination of the above points: When you combine the most difficult choices in each of the above points things often become exponentially more difficult. This is why, for example, real-time MMO games tend to be the worst, you have many players who all need to receiving information about changes in each others actions, in real-time. FPS games require high amounts of precision in the prediction and latency issues are extremely noticeable as well. Take all the debugging hassles in each category, and then debug a problem that may span events that occur using one or all of those categories and things can become messy. Real-time monitoring of events, and good tools will go a long way in making debugging easier for games.
• And more: There are probably a handful of other points as well that I just haven't thought of at the moment, such as cross-platform networking (can the client run on multiple platforms but connect to the same server?), or localization (can clients running different languages connect to the same server? If so, how would a chatbox work, how would you display Mandarin to someone on an American English client? etc.). Again, implementing these things is difficult enough, but now to properly debug those clients you need to be running in multiple languages, and multiple platforms. Can you debug a Mac client from a PC running Visual Studio, for example? Yes, but it's a pain if you've never done it.

My best advice is to use any game/networking engine that already exists and has some good documentation, reviews, and tools. Writing your own networking for anything but a simple game could take quite a bit of time, so you'll have to weigh that against what kind of time/money you have, and what kind of game you're making.

Answer 2

There are a lot of specifics that make networked gameplay much harder to debug, but in very very general terms it comes down to 2 basic points:

1. Error can arise from a much greater set of potential sources than client-only gameplay.
2. It's hard to insert debug tools into live instances, especially when huge volumes of data are being manipulated.

Usually a multiplayer game has a lot of specialized servers working together (lobby server, game logic, etc.), and many times multiple databases. An error can come from any part of this system such as a bug in your game server software, too much load on the server, incorrect server configuration, bugs or unexpected behavior in the client that send incorrect commands or data, etc. Since you can't easily use a debugger on this live code, usually most companies make use of Logging (either with their own logging tools, open-source tools, or commercial logging packages like Loggly) to debug and then also use specific tools to monitor key system performance metrics.

Keeping an eye on issues in a live environment is a MUCH more in-depth instrumentation, and requires administrators to keep an eye on constantly.

As far as design, even if you use a commercial networking package (like a commercially available networking library, game server or a PaaS/SaaS game networking tool) there's still a lot of fundamental architecture decisions you have to make for your own specific style of game (for instance, are you turn-based, real-time, %100 asynch?), no package will do that for you. So networking while engines provide tools, they don't free you from needing to know networking and network programming principles.