Skill vs. Luck, the ratio and its measurement

Question

Gamer buddies, is there a term to describe the level of variance in a game, in comparison to luck. The card game war would have 0 skill and 1.0 luck because the player cannot affect the game. I can't think of something that has 1.0 skill. At first I thought Spelling Bee, but the words chosen for each contestant is randomly chosen suggesting some luck involved... What ratios do different games have, and how can those ratios be accurately measured? What metrics could be used to accurately measure such a ratio? I'd also like to hear of any 1.0 skill games if anyone can think of one.

To reiterate the question clearly: does there exist such a measurement and if so what is it? Furthermore is there a term for the target of this measurement, so we can have a discussion using a noun.

EDIT: the term luck is used to describe the level of effect that chance, i.e. random events, have in affecting who the winner is. I appreciate everyone's responses.

Answer 1

This answer assume familiarity with normal distributions and standard deviations.

A simple but usually reasonable assumption is that we can describe the outcome of a game as a random event where player1 wins if player1's skill plus a normal distributed random variable is greater than player2's skill. The standard deviation of that normal distribution can be compared to the difference between the two player's skills, and for a larger group of players we can compare the standard deviation of the normal distribution to the standard deviation of the skill levels of that player group.

Thus if we for instance have a group of players where the standard deviation of those players's skills is double the standard deviation of the luck of the game we could with some reason say that the game for this group is 1/3 luck and 2/3 skill, but this is only valid for that specific group of players, there is no universal way to measure luck versus skill in a game.

Edit: Some examples to illustrate the difficulties of the question

All games for two players.

Flip and choose
First a coin is flipped to determine who goes first, then each player in turn choose a number from 1 to 10. Whoever choose the biggest number wins, in case of a draw the player who started wins.

Gomoku with coin flip
First a coin is flipped to determine who goes first, then the players play a standard match of Gomoku on a 15x15 board, whoever wins that game wins.

Analysis

Intuitively we'd say that Flip and choose is a game of luck, an average person would figure the optimal play before even playing a single round, so effectively the coin flip is all that matters.

Gomoku is game of skill, an average person will not be able to produce optimal play. Still, starting is an advantage so at least the flip of the coin must count for some luck in the final verdict.

With optimal play Gomoku is a win for the player who goes first, it is also a solved game, so a computer equipped with the solution database will always win if it is allowed to go first. Thus to computer players both games are trivial extensions to a standard coin flip, whoever wins the flip wins the game. This would suggest that they are both games of 100% luck. To reach any other conclusion we must consider a player base of less skill.

Answer 2

does there exist such a measurement and if so what is it?

No, no such measurement exists. While you may be able to come up with a metric for skill. You'll be hard pressed to come up with a metric for luck (unless it's controlled luck). However, the two metrics would likely be different enough that you're essentially taking the ratio of apples/oranges. Further, the metrics will vary from game to game, so comparing ratios between two games is comparing apples/oranges to GI Joes/cats.

However, there are ways to decide if a game is a game of skill or a game of chance, at least from juridical point of view. Specifically, gambling in law. A number of states in the US allow people to pay money to enter games of skill, but not games of chance (or at least significantly limit the amount of money that can be spent on games of chance). There is a paper on the topic, but the All Games of Chance website has a decent definition of how these are legally categorized:

There are two main differences between games of chance and games of skill. The first difference is who the player is playing against. When a player is playing against the house, it is a game of chance. When the player is pitted against other players, it is considered to be a game of skill. Also, if an individual can prove that a particular game involves the use of skill like strategies, statistics or math along with a factor of luck or chance, the game would be allowed and would be categorized as game of skill.

Answer 3

An important point to remember is that the importance of skill vs luck in determining the winner of a match increases as the number of games in a match increases. For example, this is why golf tournaments are 4 days long; the influence of luck (at the PGA level of play) is simply too great over a mere 18 holes.

This then provides a means of measuring the relative importance of luck vs skill: the number of matches (or alternatively, hours played) required to accurately determine the better player with a given statistical confidence. (95% would be the usual standard in such a case, as in the familiar 19 times out of 20.)

Then we get:

1. Golf would be rated at 16 rounds (of 18 holes) or 64 hours (16 rounds of 4 standard hours play) if you take the FedEx playoffs as the standard to accurately rate the players.
2. Backgammon is usually played to best of 21 I believe in tournament play, but individual games would be averaging 2 or 3 due to the doubling cube. It's rating would then be about 7 - 10 matches, but only perhaps the same 7 - 10 hours.
3. Duplicate bridge would be rated about 2 sessions of 4 hours each, looking at the elimination rounds of larger team events like Vanderbilt and Spingold.
4. Chess world championships are regularly best of 12 (and I believe Go championships are similar).

Noting particularly from the latter point, even such seminal games of skill as Chess and Go are believed to possess a considerable element of luck per individual game, when played at a professional level. This would seem to be borne out by the extreme rarity of sweeps in such competitions.

Update:
A confound when using number of hours of play is that organizing committees may have unstated reasons for extending the length of individual games. My personal belief is that the overall quality of chess games at the world level would not decrease much if the allotted time were halved. However, there seems to be the unstated intent to showcase all of the individual games as best instances of play, leading to the players having more clock time than might be strictly necessary to determine the best player. (This is not necessarily wrong, simply a complication to note when measuring relative importance of skill vs luck.)

For example, Chess and Go matches extend to an almost obscene number of hours, clearly more than necessary to determine the best player given the, both believed and evidenced, high ratio of skill to luck even in individual games. If the sole purpose of world championship matches were the determination of the best player, the number of play hours, and possibly the number of games, could be reduced for both of these games.

Answer 4

Back-of-the-Napkin approach:

1. Need a larger sample size and longer time series than you probably would suspect intuitively.
2. K.I.S.S.: How quickly do the winners and losers "revert to the mean?" If the mean "reversion/regression" is slow then skill plays a larger roll. If mean "reversion/regression" is fast then luck plays a more significant role in the outcome(s).
3. If the game is digital, and the code is locked, then trying to tease apart luck from skill is a waste of your time, since any algorithm imaginable could be shaping the outcomes.

Answer 5

Some measures have been proposed, see

• "On a relative measure of skill for games with chance elements"
• "A new relative skill measure for games with chance elements"

The basic idea from the first paper is to estimate

skill = (potential learning effect) / (potential learning effect + potential random effect)

which gives skill as a number between 0 and 1. Alas, these effects are analytically computable only for "easy" games. For a one-player game, the above equation comes down to

skill = (Gm - G0) / (Gu - G0)

where the G's are the expected net gains of three players

• '0': a beginner who plays the game in the naive way of somebody who has just mastered the rules of the game.

• 'm': a real average player who can be thought to represent the vast majority of players.

• 'u': a virtual average player whom we tell in advance (i.e. before he has to decide) the outcome of the chance elements.

As an example they calculate for American Roulette: Gu = 35 and Gm = -1/74, the latter corresponding a "simple" play (e.g. rouge/noir, pair/impair). The value for G0 is actually a matter of debate, even for this game. If the beginner goes for a simple strategy, then the skill is 0 obviously. However if G0 is for a non-simple strategy (e.g. plein, cheval, carre), then G0 is -1/37 (i.e. worse average loss.) So with the latter assumption, there is a minor potential for learning, so skill is 0.0004. I have to say I'm a little miffed that they use French terminology for the American Roulette; alas they source they cite for further details is in Dutch.

For Blackjack they derive from a computer simulation that Gm = 0.11, Gu = 27, and take G0 = -0.057 for a "mimic the dealer" strategy, and from that obttain a skill of 0.006.

For games where players compete directly and strategies like sandbagging or bluffing matter (these are the only games called multi-player games in game theory by the way), the second paper has a more sensible approach in that it considers players potentially changing strategy a source of randomness. They use the same skill formula as above (except that they calll the three types of players beginner, optimal and fictive player). The difference in their approach is that

the expected gains for player i as an optimal player are given by his expected gains in the Nash equilibrium of the related two-person zero-sum game against the coalition of the other players

and for the "fictive" player they also assume that he knows the the outcome of the randomization process of his opponents.

Alas there aren't any examples that are intersting but simple enough to relate in detail here. They compute for a simplified version of drawpoker a skill of 0.22.

Both papers emphasise however that the exact skill value depends on the definition/assumption of beginners’ behavior.

An experimental approach is needed for more complex games of practical interest, e.g.

• "The Role of Skill Versus Luck in Poker: Evidence from the World Series of Poker"

Those players identified a priori as being highly skilled achieved an average return on investment of over 30 percent, compared to a -15 percent for all other players. This large gap in returns is strong evidence in support of the idea that poker is a game of skill.