I am almost complete developing a small indie-style multiplayer game. While I intend to allow people to cheat in single-player, this is obviously not acceptable in multi-player. Does anyone know of any ways to help stop the average Joe from using something like Cheat-Engine to modify parts of the game? I currently plan to have the client upload a MD5 hash of each setting file the game uses (stored as an XML) to the game server for verification every few seconds, but is there anything I can do to stop things like memory-editors, etc?
If you are worried about locally modified code, then how can you be sure that someone hasn't simply modified your notification code to send a static list of MD5 hashes, the same ones you expect? In fact, you don't even need code modification to do this, you just need a fairly basic proxy (assuming no SSL, but even that could be faked with a bit more effort).
The only way to do what you want is to have a server and simply not trust the client at all. All calculations should be handled on the server and all actions verified as being at least somewhat possible. For instance, don't allow the client to say that they want to move to one side of the map when you know they were on the other side a moment ago. Without a server in the middle doing everything that is vital to the game, it is impossible to not have cheating, because someone smart will always find a way around anything you can build into the client. Even with one, it is still possible if you don't think through all the different ways to manipulate the situation slightly. For instance, the movement validation I mentioned before: If you do a simple point to point calculation people may still be able to teleport through walls.
So the question is, how average is 'average Joe'? You've already mentioned code editing and memory editors. That's above what I would personally consider the average level, and if you do want to be worried about that level then a separate trusted server which does all the heavy lifting is required and the client will boil down to essentially just being display and input devices.
The client is in the hands of the enemy. (The Laws of Online World Design)
Really, the only way to beat most cheats is to have the client be a "thin client", that is: To only act as an input and output device, and to never give it more information than it precisely needs.
This won't stop automation and this won't stop passive information gathering and analysis, but you can design your game such that these don't have a significant impact.
This won't stop people hacking each other via malformed messages relied by the server - your client needs to guard against messages from the server same as you would guard your web sites from SQL injection attacks.
Since you also want to use the same client for the single-player game, a solution would be to run a local server in a separate process/thread and treat it internally as a client/server setting. The "cheats" would then work on the server side. Minecraft does something similar, though it's by no means the first game to split the game engine from its interface in such a way.
In a basic multiplayer game cheating in the kind of Cheat-Engine simply won't work. The clients are only sending the data and the actions you designed it do to. Usually thats not much more than what the player "did", e.g. in what direction he is running, if he is shooting and so on. So the changes he does to the memory will only affect his own game, but the other players will see no change, a so called desync. Most games detect desyncs like that and remove the desynced player from the game.
Now there are also other ways of cheating, but those are all about how your game is designed.
To protect from the easiest hacks, which is faking network messages, the server should check those package for their sanity. "That guy just jumped 5 screens? Thats impossible, deny package." (Note that you can also go the fully synchronized route -that means everything is synchronized and only "What buttons did the player press."-kind of messages are send- which makes faking network messages useless by design.)
The last way of cheating in multiplayer are visiblity hacks, thats modifying the client so that it shows data to the player which he doesn't know. Examples for this are not showing the fog of war, being able to look through walls, displaying a minimap or other things. These are impossible to prevent.
To prevent basic Cheat Engine hacks that manipulate the values of your variables then you have to hide those values. Typically Cheat Engine is used to identify the memory location of interesting variables (say the amount of gold or life or upgrade level of an ability) by doing a search for the known value of said variable, play more of the game and cause the value to change, then Cheat Engine would do a new search from the outcome of the previous search for the new value. This allows the cheater to zoom in on the memory location of the value, now they can change the value of that memory location using Cheat Engine.
For example, I have 245 GOLD... with Cheat Engine I do a search for 245 and find many memory locations. Then I play some more and bring my gold up to 314, I then search the previous search output for the value 314 and easily find the memory location for where GOLD is stored.
The way to prevent this is to never have the real value stored in a memory location. For example, I store the value in an object that has to calculate the real value on demand when it is required. So lets say that the player has 245 GOLD. If they do a search for a memory location with the value 245, they might find many but none of them will be the memory location of where the gold value is actually stored, that is because you do not store the value 245 for gold. When the game needs to know how much gold, it will ask the object that holds the value for it, which will calculate it on demand.
So the question now is: How exactly do you store a value in a way that does not reveal it? This gets a little tricky and ugly and I'm sure there are many ways it can be done. What I like to do is store a boolean array (or byte array). The length of the array can be anything, but let's say it is 13. Then you have a counter that represents how many times does 13 go into that actual value. So if we want to represent 245 then the counter would have a value of 18. Now the array would have all booleans set to true for the remainder of 245 / 13... basically the modulus. In this case that's 11, so the first 11 booleans in the array would be set to true, the rest set to false. To retrieve the value all you need to do is multiply the counter by the array length, then add 1 for each boolean set to true (stopping at the first false). Now the number 245 would never be stored anywhere and would be hard to find the memory location that would need to be manipulated to alter the gold amount. You might want to set the array length to different sizes (maybe randomly pick a number between some reasonable range) when this object is created.
EDIT: This is useful for multiplayer and single player. There is cheating that can also be done in multiplayer, where the values in the packets can get altered. This would require different techniques to prevent, like signing each packet.