As with any security question, it is useful to think about every step in the protocol. From the user pressing a key to the server response.
Before we go into that, I must say that the only way to remove a risk is to remove the asset that is at risk. For example, the only way to make sure a cable will not break is to use a wireless connection. If we are talking about using stronger cables, hiding them from view, etc. Those are mitigations.
In your case, if you are worried that by some cheat the client may say X to the server, removing the risk means to disallow the client to say X in any circumstance. Not as a forbidden thing to say, but as something that cannot be encoded in the communication.
I am using the word mitigation a bit loosely here. If we were to talk about risk management we would talk about other forms of "mitigation" such as having a backup connection (minimizing the impact) or getting insurance (externalizing the risk), etc.
The kind of mitigation we want to focus are deterrents. To put a wild example, if somebody wanted to enter your room, a door with a lock is usually enough to stop them. However, if we were talking about a murderer decided to get you, you can imagine they could break the lock, chop the door down, etc. Therefore, the door and lock does not really remove the risk of people entering, it deters them.
Making it harder to cheat
For the solution I present the below, the idea is to make it too hard to make it not worth it.
I will go from the server to the client, because the closer we get to the user input, the crazier the schemes the attacker will have to use.
Of course, we start at the server. A compromised server is off topic for this question. Anyway, just want to say that it happens, but rarely to cheat on the game and most likely to steal credentials.
The connection from server to client is the next spot.
Aside from using a secure channel (Use a secure channel), I am more interested in the design of the protocol the client uses to talk to the server.
Consider that if the attacker can make a rogue client capable of connecting to the server, it would be free from any security part of the official client. Yet the attacker will probably not go into the trouble of creating a fully functional rogue client (instead they would prefer to modify the official one). Therefore, the intents of rogue clients are often to inject resources, items, or some form of boost into an account.
I advise to remove this risk. Do not let the client tell the server what are the rewards they get, and do not let the clients exchange items without server authorization. Be careful with the design and implementation of coupon codes (if you ever decide to have) or any similar opportunities.
You may also want to have the server authorize drops and other loot. Instead letting the client decide, have the client ask the server for drops and loot.
For example, if there is a chest and the player tries to open it, the client can request to the server the contents of the chest while it start making the animation of opening it.
Alternatively, if you may get a compromise in which the client is able to generate drops and loot, but only from an authorized set and amounts dictated by the server.
Otherwise, a rogue client would be able to create items, resources, money, etc.
That reminds me, when it comes to aim bots, one way to mitigate them is have an accuracy value that deviates the bullets. That is, the client says it shoots to X direction, but it has to ask the server if it hits. The server still may say if they miss (given an accuracy value known to the server) and how much damage it does. It sounds like the game is cheating, because it rather is, so this is not a solution for every game, it works best if shooting is secondary.
Nevertheless, aim bots are only effective because they access game memory to know where to aim. Will see below how to prevent them from tampering with game memory.
The attacker may try to alter the client on permanent storage (e.g. disk) or on ram.
3.1. Permanent Storage
Your first line of defense will be self-diagnostics: any decent online game will have a launcher capable of detecting corrupted files and patching them as part of the auto update functionality.
Aside from that, if you can afford to, add digital signatures to your executables.
The attacker can usually bypass these measures, yet it is often enough deterrent from altering the game in permanent storage. You can make it harder to bypass by using Hash and Decrypt, but ultimately not impossible, in fact, I would say it is not worth to go that length.
The attacker will just move to the next weak point. In this case, altering the program in ram.
I am outdated when it comes code injection in RAM, I will talk about the methods I know.
First, let us get the obvious out of the way: attach a debugger. It is possible to detect if your code is running under a debugger, to detect a debugger is a common anti-crack protection. To bypass it, they will have to modify the code in permanent storage... if they can find it. The usual approach is code obfuscation, some solutions will have that code under a cipher. It if is too hard, the attacker will probably not bother, they move to the next thing.
To inject code in RAM the basics are as follow: Get a handle of the game process, allocate memory for it, give executing permission for the memory you allocated, write code there (if the code you want to inject is not short, write code that will load a library with the code you want). Assuming you have root privileges, the above is possible (baring new security measures in modern OS of which I am not aware of, as I said, I am outdated), but that is not enough for that code to run on the game process.
To get it to run, you may register it as a hook, so that the OS will call it on a certain event. You may also suspend a thread, get the execution pointer, append a jump back to that position at then of the injected code, override the execution pointer to the injected code, then resume the thread. In some environment, it may also be possible to inject code to error handlers, and I am confident there are other ways that I am not aware.
To defend yourself from this:
Add a routine check that will scan all allocated pages and see they are what they should be. If you find a page you did not allocate, something is up. Enumerate all the loaded libraries, if an unknown library is loaded, something is up. There is a chance that the attacker will suspend the thread doing the check, update the expected value and then resume the thread. To mitigate that: 1) have your code run from the thread pool, 2) add a check that if the time it takes to verify is too long (presumably because the thread was suspended) something is up. Note: they may use the injected code to override your check. 3) Store the expected value as a hash or as a signature, to make it harder to tamper with.
Have a companion service or application scan the running executables. Read their code (in machine language) for calls to functions used to allocate memory in another program and to suspend threads. If it has them, consider the program to be dangerous, close the game. You may have a white list of programs known to be ok and a black list of programs not so ok. If you find a new dangerous program, have the local setup remember it. If you have the means, send a copy of the software for investigation (to you or to a security company providing this service) and potentially add it to the black list - you may have to mention that in Eula.
Note: Have your game check the companion is running and not suspended. So that the attacker cannot simply close the companion to bypass it.
Note: there are commercial Anti-Cheat solutions that package this functionality.
On the other hand, instead of trying to stop others from reading memory, we can give them a harder time using it. To do that, randomize the order in which you load information to memory in your program. This is common to prevent buffer overflow exploits. This does not make things impossible for the attacker, but it means that they will have to scan the whole memory of your game to find where they have to read the data they need - which is perfectly doable.
Going beyond memory injection, the attacker may be using virtualization to run the game, or could have made his bot a rootkit. Either way this makes it possible to mock the system calls the game makes. This gives them a proverbial back door to run code in your process. I am not aware of solutions for this... At least there is code to detect if you are running under virtualization, you will want to take the Red Pill, Neo.
Perhaps the attacker is not messing with the game at all. Perhaps all the bot of the attacker is doing is reading memory, taking screen shots (I have seen demos that use Computer Vision), and sending input.
The technique of having a companion looking for other software that may exhibit a particular behavior is useful here. In this case, you are looking for functions used to send input and messages to other applications.
I think at this point you will be in a more traditional arms race, where they come up with a new cheat, and you add it to the black list. If your game is a success to the point where cheats start to become popular, it might be worth to have an employee (again, we are assuming the game is a success) go online and search as hard as possible for any cheat for your game, try them, and add them to your black list.
Tinfoil hat on. The attacker may use custom hardware designed to cheat the game. Ok, that is enough. If they have the means, I would say they earned it.
Note: Actually, for a e-sport, in an official tournament, you want to have approved hardware. In addition, there is a point 7 that applies in that case: doping.
Coming around to deterrents, there are three ways to deter (aside from fearmongering). 1) Increase the cost of cheating, 2) increase the likelihoods of are caught and 3) increase the downside of being caught. Since I did not talk about getting cough, I will briefly go over it here.
The problem is incorporating a Turing test in the game. That is, the game must expect something from the players that only players could do. You can play this in three ways:
Make the game hard for bots: The game itself requires some skill that is easy for humans, but hard for machines. Now, be careful, because bots are good at using the correct answer once and again, even if they cannot figure it out. Therefore, this will require some level of unpredictability to increase the complexity of the tasks for the bots. Procedural generation can be useful here.
In addition, bots are also good at reading the memory and taking data from there that would not be available to humans. Your tactic should not be overwhelm the bot, because that overwhelms players too... instead use problems known to be hard. For example, bots can't play Foldit because if they could, there would have had the need to create Foldit in the first place.
Of course, this does not match every game concept, and it may be harder to implement depending on the genre you want. This approach is not for every game.
This is not exactly getting the bots caught, but making them perform badly / harder to develop.
Have a sort of Automated Public Turing test to tell Computers and Humans Apart. Sadly a straightforward CAPTCHA will not work in a game, because the player solves it and then lest the bot take control. Moreover, if you keep giving CAPTCHA to players, you annoy them.
Instead, you can analyse the behavior of players - in fact, you probably should, it is useful data - see what do they buy, how often, what gear do they use, how many hits they miss, what is their reaction time, etc. Aside from telling you what gear they prefer, and how good they are at the game, this may help you find outliers, also spot who has instant reflects and perfect aim.
There are two hurdles with this: The first one is that you cannot anonymize the data if you want to be able to do any punishment based on this. You will have to mention that in your Eula (it may have people wonder what evil things you do with their data). The second is that this is data mining on "big data". You have data, but you don't have a clear line that separates bots from humans, that is a good scenario to apply machine learning.
What behavior would you have to try to identify? See below. If successful, the bot developers would have to adapt the bots to perform worst in order to avoid detection. You will not eliminate them anyway.
Have people apply the Turing test. Machines are not good at identifying bots, but humans are. Therefore, if you turn identifying bots in part of the (meta) game you will be able to identify the bots. That is a convoluted way to say "have player denounce other players".
In multiplayer games, it is relatively easy to identify because they have the following behavior:
As soon as an enemy appears, they turn around (often even if the enemy is outside of the field of view) and shoot. With no delay, with perfect aim.
They rarely respond to chat, or socialize. For example in an MMORPG if another player comes along and salutes, they do not respond. If the other player takes drops (if your game allows for a player that did not kill to get the drop) or kills everything, the bot does not complain and does not move to another area.
They show no fatigue. They do not start to slow down, and they do not take brakes. In fact, cheaters may leave them running across the night, so they (at least the most "harmful") will be running for hours. Sometimes the cheater plays during the day, and leaves the bot at night. In fact, if an account is has the game open for many days without logging out, they are cheating or the game is taking a toll in their health. Red flag either way.
For punishment, we would probably be talking about bans, or in the worst-case account removal. I consider the punishment off topic, if in doubt; take a hint from what other games or platforms do.
Motivations of cheaters
At the point where the bot has a similar performance than human, the only reason a person have to use it is that the bot does not get tired.
Using the bot instead of playing is... well, it is not playing. One would expect that if the game is fun, a person would prefer to play. If the player doesn't like to play, why don't they play something else? Therefore, if a user is leaving the game run with the bot (even if only during the night), it raises the question: WHY?
Why would they prefer to have a bot play? Consider if your game has become tedious. Perhaps the market is suffering from inflation, prices are too high and people has to farm for too long. Perhaps you have boring parts in your game, and so people use bots to get over them to reach the fun parts.
Why would you want an aim-bot? It sounds like they do not care about the gameplay. They may only be interested in exploring the world, or opening a game option they are interested in, or know the lore of the game. Perhaps they are archivers and only want the rewards. Perhaps they are only interested in the social aspects of the game♪.
♪: Maybe they use it to meet people, maybe they want to rush the content to write
about it on the web, perhaps they know somebody who plays and want to catch up quickly, perhaps they want to rank up fast to be able to help somebody they know.
If your game is mainly competitive... Perhaps they find amusement in the reactions of players defeated by the bots. Alternatively, perhaps you are not providing players with a way to play against other players on the same skill level, resulting in new players facing up against veterans... and new players look for aim-bots to make things even.
For abstract: the game has other flaws beside the cheaters. Cheaters aggravating the situation, but are not the only problem. At some point, for somebody, the game is not intrinsically fun, and they need to resource to a bot to fix it. So, make the game fun so they do not resource to that.