How to accurately score moving averages (indepenent of sampling window)

Question

I'm developing a puzzle game, and coming up with a little trouble how to accurately create the 'score' for a particular player's solution.

In this game a player creates a machine which generates 'events'. The 'score' should be the number of events per second that a particular machine performs.

I.E. if my machine performs 6 events per second, and yours produces 8 per second, your score should be higher than mine.

I initially thought to use a moving average for this, but I've discovered it's trivially easy to manipulate this mechanic, such that solutions with an equal long-term performace don't generate the same score.

So in this case if my machine generates 1 event per second, and yours generates 3 every third second, they should have the same score. But if I use a running average to calculate the score, then the latter solution is likely to generate either a higher or lower score depending on the sampling window.

Case A:

   time    | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 |
solution A | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1  |   = 10 / 10 = 1 event per second
solution B | 3 | 0 | 0 | 3 | 0 | 0 | 3 | 0 | 0 | 3  |   = 12 / 10 = 1.2 events per second

Case B:

   time    | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 |
solution A | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1  |   = 10 / 10 = 1 event per second
solution B | 0 | 0 | 3 | 0 | 0 | 3 | 0 | 0 | 3 | 0  |   = 9 / 10 = 0.9 events per second

I'm wondering if theres some other metric possible to use other than 'average' that wouldn't be subject to this kind of inconsistency? The only way to make this works would seem to run the simulation for infinite time, at which point the two solutions would converge on 1 score / second, but that's not possible to do.

I'm somewhat doubtful that there's any solution to this problem, but just wanted to throw it out to see if there are any ideas.

Answer 1

What is it you're really looking for, a breakdown of how consistently the machine outputs events during each block of one second or how many total?

Try a running total designed to capture consistent output and downplay bursty results; you can do this by using the variability between blocks and not the total numbers.

Starting with a dummy block of 0, for each new block recorded if the new output is less than the previous block then accumulate a -1, otherwise accumulate a +1. You could also use a +- based on how big the difference is.

Using your "solution A" and "solution B" data this measure of consistency will make A look very attractive with a score of 10 while B lags behind with a score of 4.

To capture total output you really are at the mercy of your sampling window, someone will almost always be ahead or behind temporarily based on how bursty their output is and the granularity of your window compared to the total number of samples.

One defense against this behavior is to provide a margin of error to the accumulated score to counter bias caused by your window size. Using your example of 10 samples let's say that your margin of error will be 1/5. In both cases the results would be equal within the margin of error but let more significant differences prevail.