OpenGL self-test in a mobile game

Question

I've made an Android game that uses no/custom game engine. It renders using OpenGL ES 2.0 and optionally uses some extensions for optimization or extra effects. Over the time I have dealt with various issues in OpenGL drivers, I had to implement workarounds or disable usage of some features on some devices. I am still getting reports from users that something does not render correctly on their device, but in most cases I have no way to debug it as it is happening on some strange device that I don't have. I am using AWS device farm for testing and all the devices there render the game correctly...

So I have come with an idea to implement some kind of self-test. It would run shortly after the first start and render off-screen some samples that test the shaders and features I use. The images would be compared to how they are supposed to look like. (I know the comparison must have some tolerance.) If it detects a mismatch, it could send me screenshots and detailed information about the OpenGL environment. Possibly it could even self-heal by disabling some features and trying again.

Was something like that already done in a (mobile) game?

Are there any caveats I should be aware of?

Answer 1

So I have come with an idea to implement some kind of self-test. It would run shortly after the first start and render off-screen some samples that test the shaders and features I use. The images would be compared to how they are supposed to look like.

On it's own, this is a reasonable technique to verify the correctness of a rendering operation. It is vulnerable to false failures though, particularly if you implement a precise pixel-by-pixel equality test. But there are ways to make it effective, and I've seen it used on large projects as a way to do integration tests on rendering systems.

If it detects a mismatch, it could send me screenshots and detailed information about the OpenGL environment.

You will generally want to obtain permission from your users, or at least tell them that you will be collecting certain system information and reporting it back to yourself. In some cases it may be legally mandated that you disclose what you are collecting, allow a user to review it, or allow a user to opt-out of such collection.

Given that, it doesn't seem like a good use of your time to bother running this self-test on their machines. Rather you should just implement the bits that collect their system information and have them send you that when they have a bug or other rendering issue. Perhaps include some functionality to capture the current framebuffer to see the rendering issue, et cetera.

You're going to do it anyway and this way you will not be quite as inundated with reports (the bandwidth and storage of which may cost you money) of too many issues.

Possibly it could even self-heal by disabling some features and trying again.

This is potentially tricky, but a reasonable idea if done at a fairly granular level. This is often done under the umbrella of "scalability" features, although one doesn't need to actually render anything to do it. Generally detecting device capability bits and flags, or running some basic throughput tests, are all that is needed. It can be difficult to determine what you need to "turn off" based on trying to render some scene and coming up with an arbitrarily mangled output scene, for example.