Answers
- SRP - Secure Remote Password - This is based on Diffie-Hellman. The idea is that you can do a mutual password check without actually ever transferring the password or any information that can be used to derive it. Even though it's secure over the wire you should still hash and salt your passwords as your server must never store them in plain text.
- The advantage of SRP is that once it completes it also gives you a mutually negotiated encryption key that an attacker would not have been able to deduce given the data you have transferred. This means that you are free to use a symmetric encryption algorithm (like AES) once the user is authenticated.
- Assuming you are using UDP with your own reliable/ordered (connection-orientated) implementation on top of it: encrypt the entire UDP playload - including your 'packet sequence number'. If your system is designed correctly it will automatically reject replayed messages and an intruder won't be able to change the packet sequence number because it's encrypted (thus a replay is possible - but it would be automatically ignored).
Thoughts
Should your authentication be secure? Absolutely. Make no compromises in terms of security when a password is in question. Thus you should be definitely be considering the first bullet in my answer.
Should your data be secure? Only if it is an in-game purchase/micro-transaction - and then why not just use something tried and true like HTTPS. Encrypting your game traffic is not likely a viable solution for the following reasons:
- It is complete paranoia.
- It is going to add CPU time overhead on your server, unless you can purchase (expensive) hardware encryption modules.
- It doesn't matter how much security you provide for your data over the wire - someone could hijack the client process and intercept messages the moment before they are encrypted and sent. This is not only a possibility but infinitely more possible as it as substantially easier to inject code compared to intercepting packets. If you are doing this for cheat prevention you are completely and utterly wasting your time.
- In terms of password security there is unfortunately nothing you can reasonably do about a hijacked system, the client has become hostile. Blizzards WoW dongles are designed to deal with this - but I am not sure how secure that is (especially if you leave it plugged in).
Please, if you are encrypting for cheat prevention abandon it. You are going to come up short - I gave you have the information in the event that you are not. Remember that you can selectively encrypt packets by the first byte in the packet being and indicator of whether the rest is encrypted: although, once again, I would stick to HTTPS if you need to do things like credit card transactions: they are extremely infrequent and HTTPS is designed by experts - unlike something you or me designed.
All of that said, Blizzard do actually encrypt their WoW traffic. The main reason this broke is because someone, who is likely a complete amateur, decided that they would try their hand at a home-grown encryption algorithm; this panned out really well. Even if you do use industry standard algorithms there is a good chance that someone will reverse-engineer your code and simulate it - once the client enters their password there is no telling that an unsupported system is connected.
