I'm thinking about clients spoofing IP addresses, tricking other clients that they are the server; that sort of stuff. (I don't know much about this, so if this is completely wrong, please correct me.)
What should I do to prevent this? Because it is a real-time game, if I were to use encryption, I would use something fast and secure like RC4. Should I encrypt packet data with a key, that the server gives to the client?
If it makes any difference, I'm using UDP.
One possible solution is to authenticate the user using TCP+TLS; then, within the same channel, use something like Diffie-Hellman to negotiate a symmetrical key. Finally encrypt each UDP packet using a symmetrical algorithm like RC4.
You technically do not need to use TCP+TLS to negotiate the symmetrical key if you use something like SRP - just remember that clean Diffie-Hellman is vulnerable to a MITM attack.
You could go even further and use the custom SEQ field in your UDP packets (if you are using some form of reliable UDP) to implement a form of counter-mode encryption - where you add the SEQ number to the negotiated key for each packet; making it that much more difficult to mount a known-plain-text attack.
Don't let your server just hand out keys at will - a 'spoof' server could just as easily hand out it's own key; defeating the purpose of your whole encryption scheme. The only assured way is to either use TLS or mutual knowledge (such as a password/hash).
I have a distant (2 years ago) background in hacking, the hardest packets to ever crack (and what I suggest you use) is using a symmetrical key encryption method which Jonathan Dickinson described in short. You should use TCP+TLS as he also mentioned. However, he said a counter sequence.
I've run into times where a programmers "hack proof" system was easily spoofed because they have a strange enough counting system that I could crack it without programming knowledge and first year algebraic logic. As long as you pick a proper sequential method then your target should receive data exactly as expected, also meaning you should use TCP for the most secure operations.
Back on track to "in my experiences", one system I found works fantastically. A sequential method based on time sent and time expected. Since the packets must always be received in the proper order, to spoof a packet then was nearly impossible since I could never predict when a packet would be sent and when it was expected (between one packet and another) without first hacking the client program.
The short answer
In short: Each packet structure would also have a time stamp as when it was sent down to the millisecond. That's damn simple and it's really easy to check if a time is before/after another time. The reason why it makes sense so well is because the server may still receive packets in order with spoofing, without the time for authentication.
This obviously isn't the only sequential method or even the best of any method. It's just one that I've found that works very well. Combined with TCP+TLS you shouldn't have too many issues.
Personally, I wouldn't go overboard too early. Even if you encrypt your packets like Jonathan is recommending, hackers like my brother will just access the packet data before it's encrypted. If malicious users want in, they will find a way in.
Now that that's out of the way, there are ways for indie developers to minimize the damage malicious users can do. You should probably encrypt your outbound packets, but that will only stop certain kinds of attacks, don't fool yourself into thinking it's now "hack proof". To really turn hackers off, give the clients as little game-changing control as possible. One of these big MMOs out there used to allow the clients to tell the server how much XP they've earned. Guess what happened there? Don't do that. Let the client tell the server that they want to cast some super-spell on the creature, and then let the server resolve the action and then add the XP when the creature is dead. The clients should be thin, dumb, terminals that can send and receive commands, and can do some prediction (if necessary). The server should be running the game and react to commands sent by the clients.
Large game companies use the above in addition to something like VAC or PunkBuster to prevent known hackers from continuing to disrupt paying customers. How these security measures work is kept fairly secret, but I do know one method that they use: they scan currently-running applications and compare those against lists of known hacks. Once you have been caught cheating, you will be unable to join VAC/PunkBuster secured servers.
