8
\$\begingroup\$

I was wondering what peoples approaches or thoughts were on automating performance testing in XNA. Currently I am looking at only working in 2d, but that poses many areas where performance can be improved with different implementations.

An example would be if you had 2 different implementations of spatial partitioning, one may be faster than another but without doing some actual performance testing you wouldn't be able to tell which one for sure (unless you saw the code was blatantly slow in certain parts). You could write a unit test which for a given time frame kept adding/updating/removing entities for both implementations and see how many were made in each timeframe and the higher one would be the faster one (in this given example).

Another higher level example would be if you wanted to see how many entities you can have on the screen roughly without going beneath 60fps. The problem with this is to automate it you would need to use the hidden form trick or some other thing to kick off a mock game and purely test which parts you care about and disable everything else.

I know that this isnt a simple affair really as even if you can automate the tests, really it is up to a human to interpret if the results are performant enough, but as part of a build step you could have it run these tests and publish the results somewhere for comparison.

This way if you go from version 1.1 to 1.2 but have changed a few underlying algorithms you may notice that generally the performance score would have gone up, meaning you have improved your overall performance of the application, and then from 1.2 to 1.3 you may notice that you have then dropped overall performance a bit.

So has anyone automated this sort of thing in their projects, and if so how do you measure your performance comparisons at a high level and what frameworks do you use to test? As providing you have written your code so its testable/mockable for most parts you can just use your tests as a mechanism for getting some performance results...

=== Edit ===

Just for clarity, I am more interested in the best way to make use of automated tests within XNA to track your performance, not play testing or guessing by manually running your game on a machine. This is completely different to seeing if your game is playable on X hardware, it is more about tracking the change in performance as your game engine/framework changes.

As mentioned in one of the comments you could easily test "how many nodes can I insert/remove/update within QuadTreeA within 2 seconds", but you have to physically look at these results every time to see if it has changed, which may be fine and is still better than just relying on playing it to see if you notice any difference between version. However if you were to put an Assert in to notify you of a fail if it goes lower than lets say 5000 in 2 seconds you have a brittle test as it is then contextual to the hardware, not just the implementation. Although that being said these sort of automated tests are only really any use if you are running your tests as some sort of build pipeline i.e:

Checkout -> Run Unit Tests -> Run Integration Tests -> Run Performance Tests -> Package

So then you can easily compare the stats from one build to another on the CI server as a report of some sort, and again this may not mean much to anyone if you are not used to Continuous Integration. The main crux of this question is to see how people manage this between builds and how they find it best to report upon. As I said it can be subjective but as knowledge will be gained from the answers it seems a worthwhile question.

\$\endgroup\$
4
  • \$\begingroup\$ +1 great question. I haven't done this yet, but need to soon. \$\endgroup\$
    – ashes999
    Jan 18, 2012 at 12:33
  • \$\begingroup\$ Just to clarify I am not really talking about profilers or external tools really, although that could be an additional thing to help diagnose slow sections etc. What I am thinking is more about making use of your unit tests to give you some context as to if you are also improving the performance, so you could implement a new algorithm for pathfinding, and immediately test it in isolation against your previous version, and compare the numbers instantly telling you that you have improved it or wasted your time without even having to integrate it into the main project and deploy it. \$\endgroup\$
    – Grofit
    Jan 18, 2012 at 12:57
  • \$\begingroup\$ your question seems a bit confused; you're talking about general performance measuring, which can be done WITHOUT tests; but you can also write tests like "test X happens in under 3 seconds." \$\endgroup\$
    – ashes999
    Jan 18, 2012 at 14:37
  • \$\begingroup\$ Yep and that "test X happens in under 3 seconds" is along the right path, but instead thing of a test like "How many nodes can I insert into a quad tree in 5 seconds", the result of that for one build may be 10000, and the next build may be 5000. On seeing that immediately you can make an informed decision on if you have introduced a problem. The problem for me is that all this information is good, but you have to go look at it. As adding an assert for < 7500 in the time may seem fine, but if you run it on a different machine it may not pass, but in reality the IMPLEMENTATION is no slower. \$\endgroup\$
    – Grofit
    Jan 18, 2012 at 14:56

2 Answers 2

2
\$\begingroup\$

I take it you completely want to rule "Run the actual game" out, so basically my answer is disqualified from the get-go. But maybe you can take something away from it, hence why I post this regardless:

For my Master's thesis, I have various independent/parallel implementations to achieve the same thing for some modules of my game engine and I need to carry out some performance measurements. Technically, nothing would prevent me from just running the game and looking at the numbers displayed in the on-screen profiler, but I still wanted to automate that because it is a tedious process to perform whenever my implementation changes.

So what I have is this:

  • A scoped profiler (that puts an object on the stack, takes a time stamp upon construction and one upon deconstruction) that is used to measure how long the function/scope of interest took to execute
  • A module that stores a certain number of profiled samples and dumps the mean across the last n samples to a simple text file
  • An in-game command line that can be used to start a game, load a map, change the algorithm that is used within the module to be inspected, change the path of the profiler dump file and lots of other stuff. Said command line is set up to check for a certain special file within the executable directory and load it to execute the string retrieved from it (as a means of very, very crude inter process communication)

So what this allows me to do is launch my application from any half-decent scripting environment (say, windows command prompt via batch scripts - but I actually use Ruby for that purpose), set a dump file path, load a map, leave it running for a few minutes, quit the running game, set another dump file path, switch the algorithm to use, load the map again, rinse, repeat. The Ruby script communicates with the game in-flight by creating that special file the command line module is looking for and putting the desired commands in the syntax the command line understands in it.

I don't actually have use continuous integration on this project right now, but nothing would prevent me from souping up that Ruby script to also parse the generated performance logs and produce xUnit-compatible XML to communicate with the CI-system when performance has unexpectedly gone haywire for some reason and run the script on every full build on the build sever.

Alright, so neither is my game written in XNA (it's plain C++ and DirectX), nor does this approach respect the fact that you don't actually want to run the game on your build sever. It also is nowhere near as flexible as what you are probably out for - but still, it is a neat, low-tech aproach to automated performance measuring that is somewhat CI-friendly (provided one has a beefy build server).

Edit: As for how far I have actually taken that approach - I only compare performance measurings gained from different implementations of just this one module. But the whole system is set up in a way that would allow me to dump any single one of the categories defined for my lightweight profiling framework and use the external scripting environment to interpret the results in whatever way seems useful right there and then. Taking the performance profiling aspect out of the equation, I further intend to

  • Check for validity/consistency of all the assets by loading all the maps containing all the models/textures/sounds and checking the engine logs for anything unusual
  • stress-test the engine and monitor the logs for any unexpected behavior that way across a timespan of several hours/days
\$\endgroup\$
2
  • 1
    \$\begingroup\$ All good info, have given you a +1. From the sounds of it though everything you are doing above could easily be done in an integration test of sorts. The only thing you need to worry about is mocking out the actual game/simulation. As you are spot on with making your engines/frameworks components isolated so they can be tested in their own context, that is where I am trying to get at. As I am not wanting to performance test my game as that is an ever changing beast, the framework however rarely changes and can easily be setup to run a given amount of scenarios like you mention. \$\endgroup\$
    – Grofit
    Mar 19, 2012 at 8:15
  • \$\begingroup\$ Thanks. As pointed out by the things I want to achieve in the future, I aimed at automating stuff that happens in the real game - The result just so happened to be quite convenient for performance measurements as well. \$\endgroup\$
    – Koarl
    Mar 19, 2012 at 19:50
0
\$\begingroup\$

I don't see what you describe as a helpful tool. As a developer a plain performance number is almost useless.

What you want is to profile your code, split it into logical chunks of and measure how much time each of those use, peak and average. Now you can tell what part of the code is causing problems, and you know where to look for optimizations.

The tricky part is not performance changes from one build to the other, you don't need an automated to figure that. The tricky part is performance differences between different machines, there is no way to extrapolate performance from one machine to another with different video card etc.

So what you want is a benchmark feature so that you can do a one click run and get the profiling numbers. This way you'll be able to test multiple machines simultaneously. There are several ways of doing this, for instance you can override the user input to get as close to running a normal game session as possible.

You might also want to have a longer test in order to spot memory leaks.

\$\endgroup\$
8
  • 3
    \$\begingroup\$ If you follow a typical CI style approach, where you run your software through a build server you will always be testing on the same machine so always the same hardware, which gives you a baseline to your figures. As that was the direction I was coming from really. You say that you dont need a tool to figure out performance changes between builds, which is true you can run it yourself and see the difference, but it would be great if your build system or your current computer could give you those numbers without you having to do anything other than run a test. \$\endgroup\$
    – Grofit
    Jan 18, 2012 at 12:55
  • \$\begingroup\$ If you are the least bit serious about testing you need multiple different test machines. In any case, what is the actual problem? Are you unsure how to code a benchmark into the game? \$\endgroup\$ Jan 18, 2012 at 13:50
  • \$\begingroup\$ There is no problem as such, I am just trying to get some information on how some people have applied this to their projects. I don't think having separate machines helps in any way, as you are not really testing to see if it runs at 30fps on low hardware and 60fps on fast hardware. You are taking the hardware out of the equation and PURELY looking at your engine/source code. As really it shouldn't matter if you are testing on a 486 or a quad core, as you are testing one build against another, not one set of hardware againnst another. \$\endgroup\$
    – Grofit
    Jan 18, 2012 at 14:36
  • 3
    \$\begingroup\$ I have to agree a bit with both Grofit and eBusiness on this one. Automated tests are important, especially on large projects, so that when any build goes through you'll know if something has hurt or helped performance, and this is ideally done on one machine. With PC games at least you also need to test wide varieties of hardware, your automated tests may say performance is great, but then you run your game on an old GPU or find yourself running into virtual memory and all the sudden your performance tanks. You need to be able to test for those things before they reach customers hands. \$\endgroup\$
    – Nic Foster
    Jan 18, 2012 at 14:50
  • \$\begingroup\$ @Grofit The thing is, it may be just the old machine that breaks the build. It's not out of the ordinary that a change have no significant performance effect on a new computer, or is even an improvement, while the same change completely prevent the game from running on an old computer. You can't take the hardware out of the equation, there is no such thing as isolated code performance. But if you want to set up an automated test run on just a single machine, at least do it on old junker, that will give you a way better chance of the test failing. \$\endgroup\$ Jan 18, 2012 at 16:03

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .