I've made an Android game that uses no/custom game engine. It renders using OpenGL ES 2.0 and optionally uses some extensions for optimization or extra effects. Over the time I have dealt with various issues in OpenGL drivers, I had to implement workarounds or disable usage of some features on some devices. I am still getting reports from users that something does not render correctly on their device, but in most cases I have no way to debug it as it is happening on some strange device that I don't have. I am using AWS device farm for testing and all the devices there render the game correctly...
So I have come with an idea to implement some kind of self-test. It would run shortly after the first start and render off-screen some samples that test the shaders and features I use. The images would be compared to how they are supposed to look like. (I know the comparison must have some tolerance.) If it detects a mismatch, it could send me screenshots and detailed information about the OpenGL environment. Possibly it could even self-heal by disabling some features and trying again.
Was something like that already done in a (mobile) game?
Are there any caveats I should be aware of?