Automating XNA Performance Testing?

Question

I was wondering what peoples approaches or thoughts were on automating performance testing in XNA. Currently I am looking at only working in 2d, but that poses many areas where performance can be improved with different implementations.

An example would be if you had 2 different implementations of spatial partitioning, one may be faster than another but without doing some actual performance testing you wouldn't be able to tell which one for sure (unless you saw the code was blatantly slow in certain parts). You could write a unit test which for a given time frame kept adding/updating/removing entities for both implementations and see how many were made in each timeframe and the higher one would be the faster one (in this given example).

Another higher level example would be if you wanted to see how many entities you can have on the screen roughly without going beneath 60fps. The problem with this is to automate it you would need to use the hidden form trick or some other thing to kick off a mock game and purely test which parts you care about and disable everything else.

I know that this isnt a simple affair really as even if you can automate the tests, really it is up to a human to interpret if the results are performant enough, but as part of a build step you could have it run these tests and publish the results somewhere for comparison.

This way if you go from version 1.1 to 1.2 but have changed a few underlying algorithms you may notice that generally the performance score would have gone up, meaning you have improved your overall performance of the application, and then from 1.2 to 1.3 you may notice that you have then dropped overall performance a bit.

So has anyone automated this sort of thing in their projects, and if so how do you measure your performance comparisons at a high level and what frameworks do you use to test? As providing you have written your code so its testable/mockable for most parts you can just use your tests as a mechanism for getting some performance results...

=== Edit ===

Just for clarity, I am more interested in the best way to make use of automated tests within XNA to track your performance, not play testing or guessing by manually running your game on a machine. This is completely different to seeing if your game is playable on X hardware, it is more about tracking the change in performance as your game engine/framework changes.

As mentioned in one of the comments you could easily test "how many nodes can I insert/remove/update within QuadTreeA within 2 seconds", but you have to physically look at these results every time to see if it has changed, which may be fine and is still better than just relying on playing it to see if you notice any difference between version. However if you were to put an Assert in to notify you of a fail if it goes lower than lets say 5000 in 2 seconds you have a brittle test as it is then contextual to the hardware, not just the implementation. Although that being said these sort of automated tests are only really any use if you are running your tests as some sort of build pipeline i.e:

Checkout -> Run Unit Tests -> Run Integration Tests -> Run Performance Tests -> Package

So then you can easily compare the stats from one build to another on the CI server as a report of some sort, and again this may not mean much to anyone if you are not used to Continuous Integration. The main crux of this question is to see how people manage this between builds and how they find it best to report upon. As I said it can be subjective but as knowledge will be gained from the answers it seems a worthwhile question.

Answer 1

I take it you completely want to rule "Run the actual game" out, so basically my answer is disqualified from the get-go. But maybe you can take something away from it, hence why I post this regardless:

For my Master's thesis, I have various independent/parallel implementations to achieve the same thing for some modules of my game engine and I need to carry out some performance measurements. Technically, nothing would prevent me from just running the game and looking at the numbers displayed in the on-screen profiler, but I still wanted to automate that because it is a tedious process to perform whenever my implementation changes.

So what I have is this:

• A scoped profiler (that puts an object on the stack, takes a time stamp upon construction and one upon deconstruction) that is used to measure how long the function/scope of interest took to execute
• A module that stores a certain number of profiled samples and dumps the mean across the last n samples to a simple text file
• An in-game command line that can be used to start a game, load a map, change the algorithm that is used within the module to be inspected, change the path of the profiler dump file and lots of other stuff. Said command line is set up to check for a certain special file within the executable directory and load it to execute the string retrieved from it (as a means of very, very crude inter process communication)

So what this allows me to do is launch my application from any half-decent scripting environment (say, windows command prompt via batch scripts - but I actually use Ruby for that purpose), set a dump file path, load a map, leave it running for a few minutes, quit the running game, set another dump file path, switch the algorithm to use, load the map again, rinse, repeat. The Ruby script communicates with the game in-flight by creating that special file the command line module is looking for and putting the desired commands in the syntax the command line understands in it.

I don't actually have use continuous integration on this project right now, but nothing would prevent me from souping up that Ruby script to also parse the generated performance logs and produce xUnit-compatible XML to communicate with the CI-system when performance has unexpectedly gone haywire for some reason and run the script on every full build on the build sever.

Alright, so neither is my game written in XNA (it's plain C++ and DirectX), nor does this approach respect the fact that you don't actually want to run the game on your build sever. It also is nowhere near as flexible as what you are probably out for - but still, it is a neat, low-tech aproach to automated performance measuring that is somewhat CI-friendly (provided one has a beefy build server).

Edit: As for how far I have actually taken that approach - I only compare performance measurings gained from different implementations of just this one module. But the whole system is set up in a way that would allow me to dump any single one of the categories defined for my lightweight profiling framework and use the external scripting environment to interpret the results in whatever way seems useful right there and then. Taking the performance profiling aspect out of the equation, I further intend to

• Check for validity/consistency of all the assets by loading all the maps containing all the models/textures/sounds and checking the engine logs for anything unusual
• stress-test the engine and monitor the logs for any unexpected behavior that way across a timespan of several hours/days

Answer 2

I don't see what you describe as a helpful tool. As a developer a plain performance number is almost useless.

What you want is to profile your code, split it into logical chunks of and measure how much time each of those use, peak and average. Now you can tell what part of the code is causing problems, and you know where to look for optimizations.

The tricky part is not performance changes from one build to the other, you don't need an automated to figure that. The tricky part is performance differences between different machines, there is no way to extrapolate performance from one machine to another with different video card etc.

So what you want is a benchmark feature so that you can do a one click run and get the profiling numbers. This way you'll be able to test multiple machines simultaneously. There are several ways of doing this, for instance you can override the user input to get as close to running a normal game session as possible.

You might also want to have a longer test in order to spot memory leaks.