Multiplayer FPS server side performance

Question

This is related to MMO Performance except that question is about bandwidth. This is about cpu load.

I put together a simple FPS using node.js and webGL. It's extremely simple, a lot like the BuddyMaze clone of MIDI Maze. There's very little going on, everyone moves in two dimensions (no height), shoots simple projectiles, and runs into walls.

Right now, if I make multiple connections to the server where every player shoots rapidly while spinning in circles, I can get about 15 - 20 players in the game before the server maxes out a core and slows way down. And this is when running at 30 fps on the server. At 10 fps, I get about 25 - 30 connections. This is pretty bad, since the game is going to have a lot more to do soon and I'll need to fit more players for this to be a feasible endeavor.

My brother just pointed out some stats about his coworker's TF2 server. His server is lower specs than ours, yet it runs TF2, obviously a much more complex game, at a whopping 500 ticks per second, with 36 users per core. Also, we currently consume much more bandwidth than they do, but we haven't been trying to lower that much yet.

How is this possible? What sort of tricks are there to increase server performance to this magnitude? Some things that I know of include:

• Lowering framerate on the server, and interpolate positions on the client. I got some benefit, but clearly the TF2 server doesn't even bother with this.
• Doing expensive things like collision detection on the client, and verify it infrequently on the server. I haven't moved this over just yet, I will tonight. Even so I don't expect such an enormous gain.
• Break the playing field into regions (quad trees) to minimize calculations. Haven't had an opportunity for this yet.
• I've considered the unfortunate possibility that node.js is just way slower than whatever TF2 is using, and may not be suited for this kind of high intensity task.
• Is it all in the server configuration magic?

So what are the other tricks of the industry to do only the minimum required on the server but still have a flawless game experience? There's a big conflict between "defer to client to save cpu time" and "don't trust the client", so maybe it helps to know where the line is drawn in various situations?

Update

Profiling really is the only mantra I've ever found that's absolutely infallible. I quickly wrapped some timing functions around my code (thanks, FP!) and discovered what I never expected: the act of broadcasting the data to clients accounts for nearly all of the execution time. Specifically, around 90% of it. Further testing showed that this time is dependent on both the number of clients and the size of the data, but more so the latter. On a 20 user load, I cut my broadcast time 90%, from 24ms down to just over 2ms by sending only "{}" instead of the full data. But with only 5 users, broadcasting takes around 0.5 ms. So I clearly need to do some optimization here.

The first most obvious improvement is line of sight checking. This would decrease both the number of people who care about data, and also the amount of data sent to interested parties. Are there other tricks in this realm I can try, which focus on minimizing the cost of my broadcast operation?

Answer 1

Your server should not send the state of all players to all players at every tick. Instead it should send a specially crafted message to each client say every 500 ms saying "these x players in your view port should be at these coordinates in 500 ms." Most of the time this will work fine, but if the server realizes it has given wrong information it just sends an extra message.

This will decrease network traffic dramatically.

Another thing to consider is to not have game ticks on the server, but instead have the client send messages only when an action occurs (changed direction, shot fired) and then calculate ahead on the server when an action is received.

Answer 2

I've considered the unfortunate possibility that node.js is just way slower than whatever TF2 is using, and may not be suited for this kind of high intensity task.

It's probably this. TF2's server is written using C/C++, and thus, is going to be faster than node.js (which if I remember correctly, uses Javascript interpreted in Java)

Google's WebGL based Quake uses java for the server, and the source code is found here: http://code.google.com/p/quake2-gwt-port/. It might be worth looking through that to see how it's done. I also wonder what you mean when you talk about having a framerate on the server. There's no reason to render anything on the server, it should just be there for processing commands sent by the client.

Finally, the rule "do not trust the client" is more important than offloading expensive calculations onto the client in hopes of improving performance. Especially something as important as collision detection. Doubly so when your game is Javascript based, and thus fairly easy to hack (compared to something like TF2 which is compiled).

I know this isn't much of an answer, but hopefully it'll point you in a few directions that might help improve performance.