I've been interested in artificial intelligence in computer games for a long time. The question that I have been asking for a long time and to which I have not yet found an answer is the following: How to write a bot that can move through an FPS game without the game offering an API for developers? Is this possible by having only the information that the player sees on his screen? What options are there to implement something like this? Must the pixel information on the screen be processed by neural networks to operate object recognition (e.g., walls, doors, adversaries)? How can a bot, e.g. understand if he is running against a wall and is hanging on the wall? The bot would have to get a feedback about the fact that he just does not move forward in the world, because he hangs on the wall. So far I have not found an answer to these questions. For hints I would be very grateful, like also scientific articles.

As a supplement: The starting point for my question should be that you have no access to the source code of the game. Also, the game has no API through which the position data, environment data, etc. can be read.

  • \$\begingroup\$ Too incomplete for an answer, but: If the bot works on the system the game runs on, it should have access to whatever data is being transmitted to the GPU. That means it can know the worldspace vertex positions of all rendered triangles, allowing it to map the entire game world. \$\endgroup\$
    – Peethor
    Commented Jul 1, 2019 at 19:33
  • \$\begingroup\$ Are you trying to build such a bot? For which game, and to what end? Asking about the end goal you're trying to achieve can help focus answers on the aspects that are most relevant to what you need. \$\endgroup\$
    – DMGregory
    Commented Jul 1, 2019 at 19:36
  • \$\begingroup\$ @Peethor Thanks for your comment. Can you give a little more detail to your statement? I think that this data is generally not easy to obtain, since the game developers are trying hard to encrypt the whole !? \$\endgroup\$ Commented Jul 1, 2019 at 19:37
  • \$\begingroup\$ @DMGregoryIt is not a specific game at the moment. I've recently seen first-person shooters like Apex Legends and CS: GO. But also third-person shooters, such as Fortnite Battle Royale by Epic Games. I'm just interested in whether there are methodologies that can be used to write bots for such games that can behave intelligently in the world. \$\endgroup\$ Commented Jul 1, 2019 at 19:48
  • \$\begingroup\$ Cipher data that goes to the GPU? not imposible... yet, that is dumb. It implies that your shader code has to decipher. That will kill performance like hell. There could be ways to inspect the process memory to find objects, if whatever anti-cheat solution they have lets you. Besides... can you record the screen? Currently there are aimbots that use computer vision on screen captures, and computer vision solutions are more accesible than ever (OpenCV, OpenPose, TensorFlow, etc...). Go learn computer vision. \$\endgroup\$
    – Theraot
    Commented Jul 1, 2019 at 19:59

1 Answer 1


I am a little surprised that no one has answered this, and I am no expert, but the high level concept for one approach, detouring, is straightforward in concept! What your bot will do is read the graphics data, usually from the GPU, and then output keyboard and mouse mappings. You could simply use the screen as input [hard but doable] or you could use the graphics resources as input. As discussed in the machine learning paper "Playing for Data", the game communicates resources of different types, including geometric meshes, texture maps, and shaders to the gpu. The game then specifies how these resources should be combined to compose the scene. Because this is consistent between sessions, by tracking the application of resources to different scene elements, you can establish associations between these scene elements. You can intercept these communications with detouring, which is often used by screen capture software and graphics debugging and performance analysis tools. One such tool that is commonly used is RenderDoc. By implementing a wrapper for DirectX and using RenderDoc to wrap Direct3D, you can then start monitoring the graphics communication elements. With those in hand, you can decide which virtual mouse and keyboard actions to take and when.

Why don't people do this more often you might ask? Some people do, but its a lot of work, and not guaranteed to yield results.

  • 1
    \$\begingroup\$ Son, that was excatly, what I was looking for. Thank you. The provided paper was exactly what I wanted. \$\endgroup\$ Commented Apr 11, 2020 at 15:20
  • \$\begingroup\$ I should also mention, that if you end up developing a bot to play and win online games, this falls squarely within the "hacking" rules of most games and is usually strictly prohibited. Such a bot will most likely be booted. \$\endgroup\$ Commented Apr 12, 2020 at 22:54
  • \$\begingroup\$ I know. My interest is purely scientific. \$\endgroup\$ Commented Apr 13, 2020 at 12:28

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .