# How to accurately score moving averages (indepenent of sampling window)

I'm developing a puzzle game, and coming up with a little trouble how to accurately create the 'score' for a particular player's solution.

In this game a player creates a machine which generates 'events'. The 'score' should be the number of events per second that a particular machine performs.

I.E. if my machine performs 6 events per second, and yours produces 8 per second, your score should be higher than mine.

I initially thought to use a moving average for this, but I've discovered it's trivially easy to manipulate this mechanic, such that solutions with an equal long-term performace don't generate the same score.

So in this case if my machine generates 1 event per second, and yours generates 3 every third second, they should have the same score. But if I use a running average to calculate the score, then the latter solution is likely to generate either a higher or lower score depending on the sampling window.

Case A:

   time    | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 |
solution A | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1  |   = 10 / 10 = 1 event per second
solution B | 3 | 0 | 0 | 3 | 0 | 0 | 3 | 0 | 0 | 3  |   = 12 / 10 = 1.2 events per second


Case B:

   time    | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 |
solution A | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1  |   = 10 / 10 = 1 event per second
solution B | 0 | 0 | 3 | 0 | 0 | 3 | 0 | 0 | 3 | 0  |   = 9 / 10 = 0.9 events per second


I'm wondering if theres some other metric possible to use other than 'average' that wouldn't be subject to this kind of inconsistency? The only way to make this works would seem to run the simulation for infinite time, at which point the two solutions would converge on 1 score / second, but that's not possible to do.

I'm somewhat doubtful that there's any solution to this problem, but just wanted to throw it out to see if there are any ideas.

What is it you're really looking for, a breakdown of how consistently the machine outputs events during each block of one second or how many total?

Try a running total designed to capture consistent output and downplay bursty results; you can do this by using the variability between blocks and not the total numbers.

Starting with a dummy block of 0, for each new block recorded if the new output is less than the previous block then accumulate a -1, otherwise accumulate a +1. You could also use a +- based on how big the difference is.

Using your "solution A" and "solution B" data this measure of consistency will make A look very attractive with a score of 10 while B lags behind with a score of 4.

To capture total output you really are at the mercy of your sampling window, someone will almost always be ahead or behind temporarily based on how bursty their output is and the granularity of your window compared to the total number of samples.

One defense against this behavior is to provide a margin of error to the accumulated score to counter bias caused by your window size. Using your example of 10 samples let's say that your margin of error will be 1/5. In both cases the results would be equal within the margin of error but let more significant differences prevail.

• Thanks for your feedback, some interesting things to think about. I'm not necessarily going for consistency, as in there's no particular reason why I would favor solution A over solution B in my example. I guess to perfectly sum up the mechanism I'm looking for is: "If I ran the simulation for infinite time, what would the score per second value converge to?" The margin of error is an interesting idea also, though I can't rule out for sure if someone might come up with a solution that does 11 events every 11th cycle, thus making this hard to calibrate. ...continued... – Tim Apr 25 '12 at 23:52
• I know that there realistically there is some upper limit on the "burstyness" that a particular design could generate, though it would be tough to accurately figure that out beforehand in every case (it's kind of an open ended puzzle). – Tim Apr 25 '12 at 23:53
• @Tim I totally understand what you're trying for. You're bumping up against sampling theory and the margin of error is just a hacky way to get around not having a proper sample; even in pure statistics everything is done with "confidence intervals" which is a fancy way of embodying "margin of error." I don't know about the rest of your game design but you could make time limits intrinsic to the puzzle, forcing it to be part of the problem the players grapple with and made obvious in the descriptions and GUI. – Patrick Hughes Apr 26 '12 at 0:11