How do game bots perceive the game world & other entities?

Question

This question has been on my mind for a while...mainly because I see bots for all sorts of games like WoW and others. My question is; how do the bots know what is appearing on the screen? I don't play WoW so my example may be wrong but if for example there is a monster, how does the bot know where that monster is on the screen and how does it know how to interact with it?

Can you apply this to any game or is it specific for each game? I'm sorry if the question isn't clear...and I'm not asking how to make a bot, more asking how they detect things on the screen as its quite fascinating to me!

Thanks in advance :)

Answer 1

There are many points where a bot can inject itself into the game.

• The screen is one of them, but by far not the most useful. However, I have once seen a very early aimbot for Counter Strike which used color coding. It came with alternative character models with single-colored textures (the game was modding-friendly enough to allow this) and then just detected pixels of those colors. Not a very effective method, though. It was already quite clunky back then, and becomes less and less viable because graphic engines become more and more powerful, which means more and more detail to confuse any optical recognition algorithms.

• Another point is reading the memory directly^[1]. It is possible to have one program on your computer read the memory of another. So the developer just needs to find out at which memory address the game is saving the information which is relevant to the bot. There are tools which assist the developer with finding what they want by creating a memory image and providing various search tools. A countermeasure is to use address layout randomization, but a smart bot might be able to still find what they are looking for automatically.

• It is possible to modify the game executable itself. In order to do this the bot developers need to be able to read and program in assembler, which isn't that hard with some practice. They then look for the code which handles the information they are interested in and rewrite it to pass it to their bot. A possible countermeasure from the perspective of the game developer is to use an obfuscator to make the game's assembler code less readable, but these are usually not good for performance and there are tools for many obfuscators which reverse their work.

• And then there is the netcode. In an online game, the server sends the properties and positions of all objects in the game via network. The network data stream between server and client can be intercepted and analyzed. A popular tool for doing this is wireshark. When the developer reverse-engineered the netcode, they can write a bot which intercepts the network traffic just like Wireshark does and uses the information to make its decision. When the developer is more motivated, they can even develop a complete game client from scratch which implements the network protocol and plays the game without even having any graphical output. Such bots are very popular with gold farmers because without the graphics output the client is usually far more resource-friendly, which means they can run a lot of them at once on a small server.

---

^1: Since you mention WoW, it's worth pointing out that the early and notorious WoWGlider botting program used direct memory access. Blizzard had a subprogram called Warden that was designed to detect and block outside programs from accessing WoW's internal game state. Since the bot program circumvented this protection and read the memory without permission, the 2006 lawsuit MDY v Blizzard was decided in favor of Blizzard on the grounds that the bot maker was encouraging and enabling its users to commit copyright infringement and violate the DMCA.

Answer 2

Bots are players. They just read the player's/monster's positions from the server just like every other player does and use those values combined with a simple AI script to perform actions in the game by simulating keyboard strokes and mouse clicks programmatically. If bots actually had an algorithm to detect what monster is on the screen and interact with it, that would be a quite impressive piece of software which would worth millions. That's basically AI for robots.

Answer 3

Phillip already gave a great overview of how how most bots operate, but I just wanted to cover things in a little more detail, as I have a bit of personal experience developing on or for all the types of bots he covered.

In Runescape, there was a large project (RSBot) that would copy the contents of memory from the Runescape client into its local memory, where it could then view the entire state of the game with no risk of the client catching on. It required a bit of reverse engineering to determine where in memory to look in order to find the pointers to the data, but once they had done so, they exposed an API to take advantage of the information. It would know exactly what was where in the world by getting the coordinates of an object and then transforming them with the camera transform matrix to get the on-screen position. The hit masks were also readable, so it was trivial to determine exactly what range to move the mouse into in order to get a desired result.

The bot provided a bunch of debugging information, such as annotations that told the developer which tile coordinates are where, which ID this object has, what ID belongs to a given item, etc. This information could then be used to make bots. The actual process of making the bot scripts was actually quite simple. The framework provided many utility functions, such as move_to(world_coordinates) or mouse_move(x,y) which would perform the specified actions in a somewhat believable way (moving the mouse along a random spline, repeatedly moving via both the minimap and screen, and so on)

Also in Runescape, you have the option of drastically reducing the quality of the graphics. It was actually pretty easy to make certain kinds of bots by filming the screen and applying some basic computer vision concepts in order to construct a model of the world. I made both a curse bot and smelting bot using this technique, both of which worked quite well. It would just take the frame, increase the saturation as much as it could, and then try to extract patterns from it which it could then generate a probability map for click zones.

In the case of the curse bot, the target was a lesser demon, which is just a huge red thing, which is trivial to find. In the case of the smelting bot, it would look for a small orange trapezoid surrounded by grey, and that would be the furnace. It would also try to find a large, light-grey 'L' shape, which would be the bank desk. That was just done with downscaling and basic heuristics. It could also orient itself with the handy compass next to the minimap, so it could orient the camera in a more reliable position to find the target objects.

In various Nexon games, the servers are very trusting of the clients. I abused this a ton in Maplestory by modifying maps to remove hazards or add warps so that a bot doesn't have to be nearly as sophisticated. I would also monitor the traffic to and from the server in order to build a model of the world which could then be used to quickly find and kill monsters.

Once I realized that I didn't actually play MMOs anymore and just automated them, I decided to stop playing altogether. If I wasn't as worried about exposing myself legally, I could have probably made a killing selling MMO bots, but I really didn't want to get sued. That is why I spend my time doing real programming these days!

Answer 4

Phillip's answer is great, but just a quick note about the Model/View/Controller pattern, or MVC for short.

Your screen is part of the "view", which presents data from the "model", which is the real state of the game if you like. The bots, and your character, all 'exist' (in a sense) within the model. Imagine if you unplugged your screen - the game would continue even though the view has gone.

In a networked game, the concept of a "model" is a little complicated, as you actually have lots of models. You have the model on the server, and then a model on each client, which updates itself in part due to data it receieves from the server. You could think of the "model" as encompassing the server's model and your client's model, or you could think of it as just your own client's model. Either way, this is where the bots live. So, their "perceptions", which we could use as a shorthand for their data input and state, are based on the state of the model at that time and the interaction between the different entities/objects within that model. In practise, each bot might have a function called "refreshState" which runs once per frame or something, and which causes it to update it's own state by looking up some data from inside the model, which in turn might affect the decisions it makes.

Answer 5

They can read the coords from a file or a server but also with a ray collision detection system. It is common to use rays that goes from the bot(in that case) to outside him. If the ray collides with something then it returns some information to the bot such as if what the ray found was a player, a bot, an animal or a simple wall. That is because in the case of the player and the bot for example they have an invisible capsule that involves them entirely with information about its type for example (human, bot, ...). With that they can avoid crashing with walls and other elemnts if they are following you for example.

You can see the part Collision detection of this Page