4
\$\begingroup\$

Let's say we run a tournament for 10 bots playing 1-1 fights in card game with significant randomness influence. To measure bot skills level we run round-robin with 10k matches per each pair. Ranking shows that bots #3, #4 and #5 have very similar result.

Some time later 11th bot is developed. What is the best strategy to rank it honestly?

  • Re-run whole tournament? Bots with similar results can come out on different positions.
  • Run all ranked bots against new one? Again, those results could re-rank previous positions, if taken into account.
  • something different?

CPU time is not limited and tournaments are run offline.

\$\endgroup\$
1
\$\begingroup\$

This sounds a lot like a problem in Genetic Algorithm with AI Natural Selection.

But if this game is a tournamen, it depends on how you approach it.

For example. If you do the tree tourney style, where you climb up a tree to the very top, obviously you are going to have problems.

However, if you run it through an array set up where everyone fights everyone. You can get better results. Think of Yugi-Oh and Magic tournaments.

The number of times the fights have been made can also have a drastic effect on results. Best of three, best of four, best of five, etc.

Then... if lower level placing actually matters. You can have them dual one another for their places.

\$\endgroup\$
0
\$\begingroup\$

If rank is a normalized score, you can run all ranked bots angainst the new one. If not you have to ask why the score was not normalized and then re run the tournment.

\$\endgroup\$
  • 2
    \$\begingroup\$ What does "normalization" mean in this case? \$\endgroup\$ – tomash Nov 28 '14 at 6:19
  • \$\begingroup\$ If the score does not depend upon the number of matches, it is normalized, if it depends linearly it, then it can be normalized. However you can do a simulation using the data you already have by recomputing the scores ignoring the matches that involve a choosen bot of your. \$\endgroup\$ – FxIII Nov 28 '14 at 8:38
  • \$\begingroup\$ Problem in my question is randomness - in one 10k run bot#3 can win 48% matches, but in second - 46%. Shall I run bigger batches to reach some established confidence level for each duel? \$\endgroup\$ – tomash Nov 28 '14 at 10:35
  • \$\begingroup\$ @tomash if your variance due to randomness is that little, I'm not sure if you really care. It seems pretty stable already. \$\endgroup\$ – ashes999 Aug 24 '15 at 3:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.