Is it possible to record video, depth map and 6D pose of any games?

Question

I'm not a game developer, but a researcher working on improving video quality in various contexts. For my current problem, I need to collect videos which have both RGB data as well as depth data. Video can be monocular or stereo. However, if there is egomotion, I'll need the camera pose (rotation and translation) as well. Is there any way to record all these information in any kind of games? Games can be on any platform: PC games, mobile games or VR games.

In short, I want to record gameplay but with additional information such as depth map and camera pose.

Answer 1

One solution is to use DLL injection. This technique facilitates attaching and executing a payload (your code) within the address space of another process (the game or graphics driver) by way of a dynamic link library (DLL).

Once the injected, your payload code would:

• intercept instructions of interest
• process them as needed (log them for later, perform calculations, etc)
• pass the instructions along to the intended recipient (allowing gameplay to continue)

This technique was used in the paper "Playing for Data: Ground Truth from Computer Games" by Richter et al. and in Matthew Fisher's research on Starcraft 2 AI. You can read their work and code for more details.

Caveat: Some developers take significant steps to detect & deter DLL injection as this same technique is sometimes used for the purposes of cheating. Furthermore, some developers attempt to limit the ways their software can be used via terms of service, user license, etc. Proceed accordingly.