22
\$\begingroup\$

I've been embracing Test-Driven Development (TDD) recently and it's had wonderful impacts on my development output and the resiliency of my codebase. I would like to extend this approach to some of the rendering work that I do in OpenGL, but I've been unable to find any good approaches to this.

I'll start with a concrete example so we know what kinds of things I want to test; lets say I want to create a unit cube that rotates about some axis, and that I want to ensure that, for some number of frames, each frame is rendered correctly.

How can I create an automated test case for this? Preferably, I'd even be able to write a test case before writing any code to render the cube (per usual TDD practices.) Among many other things, I'd want to make sure that the cube's size, location, and orientation are correct in each rendered frame. I may even want to make sure that the lighting equations in my shaders are correct in each frame.

The only remotely useful approach to this that I've come across involves comparing rendered output to a reference output, which generally precludes TDD practice, and is very cumbersome.

I could go on about other desired requirements, but I'm afraid the ones I've listed already are out of reach.

\$\endgroup\$
5
  • 3
    \$\begingroup\$ The entire problem here is that your test case is determined by a good output; but what happens if that output is buggy (false positive)? Either way I would start by checking wireframes first; then move to a forward-rendered scene and finally deferred (if you use that). You can XOR the entire image to make a quick comparison (entirely black passes). The GPU is really a bad area to apply TDD; but if you come up with anything smart I would love to know. \$\endgroup\$ Jan 3, 2012 at 21:01
  • \$\begingroup\$ I suspect that I won't get exactly the answer I hope for, but I still hope for some good ideas. Good thought on black pass. Testing the depth buffer could also be helpful. \$\endgroup\$
    – notlesh
    Jan 3, 2012 at 21:26
  • 2
    \$\begingroup\$ aras-p.info/blog/2007/07/31/testing-graphics-code \$\endgroup\$
    – Adam
    Jan 3, 2012 at 21:55
  • \$\begingroup\$ @Adam thanks for sharing that. Unfortunately, it's way out of reach for an indie like me working on mobile games :) It also fails most of my basic requirements. \$\endgroup\$
    – notlesh
    Jan 3, 2012 at 22:10
  • \$\begingroup\$ @Adam you should definitely 'answerize' that link. Quite novel. \$\endgroup\$ Jan 4, 2012 at 9:12

4 Answers 4

9
\$\begingroup\$

This seems like a good application of the Approval Tests framework or something like it.

As stated in the comments, you are still going to have an issue with false positives, if you happen to approve bad output, but this will at LEAST tell you when output has changed significantly.

Since you are using OpenGL, I'm assuming that approvals won't work for you directly, but the idea is sound. Just check the file hash and if it is different, show the failure in an appropriate diff viewer (like an image diff program). If you approve of the new version, update the "approved" file to match the new result.

\$\endgroup\$
4
  • \$\begingroup\$ That's an interesting tool. If you took something like the testing farm in Adam's link in the comments above, and applied an integrated approval mechanism like this, you'd probably start to approach something that is pretty low overhead. Thanks for sharing! \$\endgroup\$
    – notlesh
    Jan 4, 2012 at 6:29
  • \$\begingroup\$ The approval test is not automated test. \$\endgroup\$
    – zwcloud
    Oct 11, 2018 at 1:44
  • \$\begingroup\$ @zwcloud: What do you mean? It's automated once it is authored, just like any test. It's just that the first approval is part of the authoring process. \$\endgroup\$ Oct 31, 2018 at 17:07
  • \$\begingroup\$ @JohnGietzen I failed to understand that from your answer. I think you should clarify the idea of Approval Tests more explicitly. I will remove the downvote when the answer is updated. \$\endgroup\$
    – zwcloud
    Nov 4, 2018 at 2:23
7
\$\begingroup\$

I'm not into the game business, so please bear any potential stupid and naive perception

For me, writing tests that compare fully rendered images are not really unit tests, but full integration tests, because everything has to work fine for a successful test run.

What about an intermediate layer where you can check that everything is fine? There are two things that come to my mind:

  1. Don't draw directly, but issue a command stream that is turned into rendering calls down the way. The command stream can be checked for correctness. This is no end-to-end test, but you want unit tests, not full integration tests.

  2. This is really a variation on 1. Wrap OpenGL, catch all calls to it, strip this down to what you really want to test, and check if output still matches. The OpenGL wrapper can do the check itself if properly configured, and turns into a mock.

\$\endgroup\$
4
\$\begingroup\$

The problem is that 3d engines aren't required to output an exact image bit-for-bit. Some leeway is tolerated as long as the artifacts caused by it aren't visible to the viewer. A texture being 1% darker than expected is usually not a serious problem... usually. Under some conditions it might be a visual artifact. And that's the problem: It is hard to judge for an automated test which artifacts would be noticed by a human viewer and which are not. Still, automated tests can be used for simple sanity checks when used on very simple geometry.

What you could do is render some simple scenes and then compare the generated image with a reference rendering. That reference rendering could either come from another graphic engine or be an earlier screenshots from the same engine (regression testing).

When some tests fail after a change to the graphic engine, a human tester should compare the output to the reference image and judge for themself if the new output looks just as good as before (or even better). When that's the case, the test should be changed to compare to the new, improved outputs.

Another thing you could check automatically are performance requirements. One such requirement could be that during a self-running demo (simulating a real sequence from the game) the frame-rate must not drop below 40 Fps on the test system. This is something you can test automatically. What you can't test is that the rendering is satisfying. But you can use such tests to prevent your developers from developing a game which looks superb but doesn't run properly on any affordable system until years after launch (hello Crytek).

\$\endgroup\$
0
\$\begingroup\$

After creating an OpenGL context (e.g. via Pbuffer) you can render a quad to a framebuffer using a floating point texture with a single pixel as a color attachment. You then render the output of a function call to the pixel in the fragment shader where the fragment shader calls the function and is linked with a code fragment containing the function. I.e. for each test a fragment shader with different method parameters is generated (e.g. using a templating library). glGetTexImage can be used to read out the pixel for comparison in the unit test.

\$\endgroup\$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .