Let's say we run a tournament for 10 bots playing 1-1 fights in card game with significant randomness influence. To measure bot skills level we run round-robin with 10k matches per each pair. Ranking shows that bots #3, #4 and #5 have very similar result.
Some time later 11th bot is developed. What is the best strategy to rank it honestly?
- Re-run whole tournament? Bots with similar results can come out on different positions.
- Run all ranked bots against new one? Again, those results could re-rank previous positions, if taken into account.
- something different?
CPU time is not limited and tournaments are run offline.