How does one test the effectiveness of a game engine?

Question

A few friends and I are working on a home-brewed engine for a selection of games we wish to produce in the future. We're making it to satisfy a few key requirements that we haven't found were properly handled in professional game engines. But we were wondering, how can we test how effective our engine is at solving the problems we set out to solve?

For example, our games require a very specific internal OOP architecture, and the engine facilitates that, making it easy for us to do what we want. How are we supposed to make sure the engine actually helps us?

The reason I ask is because I've written unit tests before, but they've usually been to test whether or not something is correct. Does the engine properly determine collisions between convex polygons? Does the event propagator properly respond to user input? Is the implementation of Bresenham's line algorithm for rendering line primitives correct? Things like this.

When it comes to testing effectiveness of the application as a whole, however, there is no notion of right or wrong. It's fuzzy, it could be pretty effective, very effective, or not effective at all. Is the only solution just to make a game using the engine, and determine the engine's effectiveness from there? I want to say no, because the point of testing is to catch errors early on right, so if we make a game to test effectiveness, we have to build the entire engine beforehand.

How would tests like these even be included? If we integrated "test games" using the same program we use to integrate unit tests, they'd get run every time the engine builds. But we don't want our stupid test games to start running every time we build.

So is there another solution?

Answer 1

Engineering is all about having to prototype machines in order to see whether a concept works. There is some minimum amount of labour required to test any such theory in working form - usually quite a lot - and until that labour's been put in and the system is in working form, it is just a theory.

An engine is largely complex (especially in the temporal sense) and highly specific in its operation, and since it exists to provide a runtime environment, it also dictates all the rules of that environment. For this reason, there's no better way to test the engine than to see if it adequately supports the game specifications intended to run within its environment. The engine is the scaffold; the tests are the games or simulations that run therein. And more cases = better coverage.

In the very first place then, to know whether your engine meets your standards, you must know what those standards are. Until then, the question "is it good enough" cannot be answered definitively. Recall the idea of why we cannot use traditional testing to test gameplay - again, in the gameplay arena our logic is rather fuzzy so we do not have (thankfully) a pre-specified set of requirements that makes a game "enjoyable" vs. "not enjoyable". The less fuzzy, the more testable.

Re traditional testing, this tends to be used for well-encapsulated, low-level, easily judgeable features like, "did this texture load correctly" or "when rendering, does this font display at the specified height" as well as things like "can we maintain N frames per second over one hour". (Visual aspects, like font display, may require tests to do things like take screenshots of the framebuffer and take measurements from there, but that's a whole other can of worms.)

And you're right, the typical way to test runtime modules would be to have a separate script that runs these perhaps triggered only on certain types of builds or only when you explicitly execute it.

Also see these rather nice anecdotes from those who've implemented automated testing in games.