I've been interested in artificial intelligence in computer games for a long time. The question that I have been asking for a long time and to which I have not yet found an answer is the following: How to write a bot that can move through an FPS game without the game offering an API for developers? Is this possible by having only the information that the player sees on his screen? What options are there to implement something like this? Must the pixel information on the screen be processed by neural networks to operate object recognition (e.g., walls, doors, adversaries)? How can a bot, e.g. understand if he is running against a wall and is hanging on the wall? The bot would have to get a feedback about the fact that he just does not move forward in the world, because he hangs on the wall. So far I have not found an answer to these questions. For hints I would be very grateful, like also scientific articles.

As a supplement: The starting point for my question should be that you have no access to the source code of the game. Also, the game has no API through which the position data, environment data, etc. can be read.

  • \$\begingroup\$ Too incomplete for an answer, but: If the bot works on the system the game runs on, it should have access to whatever data is being transmitted to the GPU. That means it can know the worldspace vertex positions of all rendered triangles, allowing it to map the entire game world. \$\endgroup\$
    – Peethor
    Jul 1 '19 at 19:33
  • \$\begingroup\$ Are you trying to build such a bot? For which game, and to what end? Asking about the end goal you're trying to achieve can help focus answers on the aspects that are most relevant to what you need. \$\endgroup\$
    – DMGregory
    Jul 1 '19 at 19:36
  • \$\begingroup\$ @Peethor Thanks for your comment. Can you give a little more detail to your statement? I think that this data is generally not easy to obtain, since the game developers are trying hard to encrypt the whole !? \$\endgroup\$ Jul 1 '19 at 19:37
  • \$\begingroup\$ @DMGregoryIt is not a specific game at the moment. I've recently seen first-person shooters like Apex Legends and CS: GO. But also third-person shooters, such as Fortnite Battle Royale by Epic Games. I'm just interested in whether there are methodologies that can be used to write bots for such games that can behave intelligently in the world. \$\endgroup\$ Jul 1 '19 at 19:48
  • \$\begingroup\$ Cipher data that goes to the GPU? not imposible... yet, that is dumb. It implies that your shader code has to decipher. That will kill performance like hell. There could be ways to inspect the process memory to find objects, if whatever anti-cheat solution they have lets you. Besides... can you record the screen? Currently there are aimbots that use computer vision on screen captures, and computer vision solutions are more accesible than ever (OpenCV, OpenPose, TensorFlow, etc...). Go learn computer vision. \$\endgroup\$
    – Theraot
    Jul 1 '19 at 19:59

I am a little surprised that no one has answered this, and I am no expert, but the high level concept for one approach, detouring, is straightforward in concept! What your bot will do is read the graphics data, usually from the GPU, and then output keyboard and mouse mappings. You could simply use the screen as input [hard but doable] or you could use the graphics resources as input. As discussed in the machine learning paper "Playing for Data", the game communicates resources of different types, including geometric meshes, texture maps, and shaders to the gpu. The game then specifies how these resources should be combined to compose the scene. Because this is consistent between sessions, by tracking the application of resources to different scene elements, you can establish associations between these scene elements. You can intercept these communications with detouring, which is often used by screen capture software and graphics debugging and performance analysis tools. One such tool that is commonly used is RenderDoc. By implementing a wrapper for DirectX and using RenderDoc to wrap Direct3D, you can then start monitoring the graphics communication elements. With those in hand, you can decide which virtual mouse and keyboard actions to take and when.

Why don't people do this more often you might ask? Some people do, but its a lot of work, and not guaranteed to yield results.

  • 1
    \$\begingroup\$ Son, that was excatly, what I was looking for. Thank you. The provided paper was exactly what I wanted. \$\endgroup\$ Apr 11 '20 at 15:20
  • \$\begingroup\$ I should also mention, that if you end up developing a bot to play and win online games, this falls squarely within the "hacking" rules of most games and is usually strictly prohibited. Such a bot will most likely be booted. \$\endgroup\$ Apr 12 '20 at 22:54
  • \$\begingroup\$ I know. My interest is purely scientific. \$\endgroup\$ Apr 13 '20 at 12:28

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .