How to develop a bot for a(n) (arbitrary) FPS game without a game API?

Question

I've been interested in artificial intelligence in computer games for a long time. The question that I have been asking for a long time and to which I have not yet found an answer is the following: How to write a bot that can move through an FPS game without the game offering an API for developers? Is this possible by having only the information that the player sees on his screen? What options are there to implement something like this? Must the pixel information on the screen be processed by neural networks to operate object recognition (e.g., walls, doors, adversaries)? How can a bot, e.g. understand if he is running against a wall and is hanging on the wall? The bot would have to get a feedback about the fact that he just does not move forward in the world, because he hangs on the wall. So far I have not found an answer to these questions. For hints I would be very grateful, like also scientific articles.

As a supplement: The starting point for my question should be that you have no access to the source code of the game. Also, the game has no API through which the position data, environment data, etc. can be read.

Answer 1

I am a little surprised that no one has answered this, and I am no expert, but the high level concept for one approach, detouring, is straightforward in concept! What your bot will do is read the graphics data, usually from the GPU, and then output keyboard and mouse mappings. You could simply use the screen as input [hard but doable] or you could use the graphics resources as input. As discussed in the machine learning paper "Playing for Data", the game communicates resources of different types, including geometric meshes, texture maps, and shaders to the gpu. The game then specifies how these resources should be combined to compose the scene. Because this is consistent between sessions, by tracking the application of resources to different scene elements, you can establish associations between these scene elements. You can intercept these communications with detouring, which is often used by screen capture software and graphics debugging and performance analysis tools. One such tool that is commonly used is RenderDoc. By implementing a wrapper for DirectX and using RenderDoc to wrap Direct3D, you can then start monitoring the graphics communication elements. With those in hand, you can decide which virtual mouse and keyboard actions to take and when.

Why don't people do this more often you might ask? Some people do, but its a lot of work, and not guaranteed to yield results.