1
\$\begingroup\$

I've created two simulations using Unity that perform the same functions, but using two different programming paradigms (object-oriented and data-oriented) to demonstrate the performance benefits of one paradigm over the other. I've collected CPU speed values (which measures the time between each loop in the simulation) for both simulations. I've compiled two arrays that contain these CPU speed values for 300 iterations of the simulations and I want to perform a statistical analysis on these two arrays to determine whether there is a significant difference in performance between the simulations.

My current issue is that I am not sure how to classify my data. My best guess is that my data is parametric (and I'm really unsure about that) and that I should be using an independent t-test, but I've also read a few things about the Kruskal Wallis and the Mann Whitney that make me question whether or not the t-test is the appropriate avenue for approaching this problem.

Any help would be much appreciated! I'm not much of a stats guy, so I'm not very confident in this aspect of game development research.

\$\endgroup\$
4
  • 1
    \$\begingroup\$ I'm not sure game developers can help with this statistical problem. Maybe check with stats.stackexchange.com \$\endgroup\$
    – Almo
    Commented Mar 22, 2022 at 1:54
  • \$\begingroup\$ Probably a better call. I figured I'd ask here first since the problem related to game performance and I'd imagine there are more people besides me doing research related to game dev. Thanks! \$\endgroup\$
    – Mav
    Commented Mar 22, 2022 at 1:59
  • \$\begingroup\$ Remember that not every question that involves a game is a question that game developers are best positioned to answer. If you need help with the stats part, ask a statistician. Ask here when you need help with the game part. \$\endgroup\$
    – DMGregory
    Commented Mar 22, 2022 at 3:18
  • \$\begingroup\$ @BrettMaverickMartin The usual way to do this in game development is to look at the FPS counter and check which one is higher. When there is no clearly visible difference between two solution, then the performance difference doesn't matter enough to outweigh other concerns. \$\endgroup\$
    – Philipp
    Commented Mar 22, 2022 at 9:06

1 Answer 1

3
\$\begingroup\$

To answer the question, in case anyone else finds themselves in a similar situation, the correct test in this particular case is a Wilcoxon Rank-Sum test. I determined that my data was non-parametric, after which is just the non-parametric version of the independent t-test.

\$\endgroup\$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .