How can I know how difficult a question is?

Question

I'm making a "Who Wants to be a Millionaire" style of game. I'm really stuck at the part where I should arrange questions from easy to hard.

Isn't this subjective? How can I tell how hard/difficult a question is?

Answer 1

Your assumption that question difficulty is subjective and is itself a hard question is absolutely correct! From a professional assessment standpoint it's a known issue, and while there are techniques to assist, it is best considered "an ongoing process" that requires testing and adjustment - like trying to figure out what just the right amount of salt to add to a soup.

However, there is some good news - in game shows (and their video game equivalents), they don't actually do as much ranking by difficulty as you think. In Jeopardy games, as a for-instance, sometimes $1000 questions will reappear in other play-throughs as $100 questions! So let's consider how Who Wants To Be A Millionaire-style games do it.

First, these style of games have changed a lot over time, so I'll go with the "Classic Millionaire" approach that the show started with. As the questions move up, it's better not to think of a strict rank-ordering as in each question has a specific class, but rather that they are more broadly grouped.

The early sets of questions (say, the first 2-4) were written with children in mind. This was to give a chance to let families play, and also to give the contestant a chance to warm up/adjust to their nervousness. The questions might be inspired by actual school child homework, or just generally made easy to understand (no big words), not require special deep knowledge in any particular subject, and 1-2 of the 4 multiple choice options were pretty obviously silly (usually with one trying to be outright funny).

Once the easy questions are done, the questions are more challenging. Challenge is usually defined as a mixture of these three elements:

• The question requires special, specific knowledge that is not commonly known (who is the 17th President of the United States?). Popular trivia might require you to know the binomial nomenclature for specific plants and animals, or of a particular part of a certain places history.

• The multiple choice options presented. Are any of them specifically recognizable as definitely not right? Can you logically exclude some possibilities? If the question was about an early US President and one of the choices is someone you don't think was president ever, or if they were but much more recently, then you can narrow down the options.

• Can an answer be logically arrived at, or do you just have to know the answer outright? If you are asking about a botanical name and all the options are botanical names, then you either know the answer or you don't - a genius IQ and critical thinking is of no help if "if you either know it or you don't".

So the total question difficulty is related both to how obscure the trivia being asked is (rarity of knowledge), whether or not the question is pure trivia or if you can at least partially reason an answer logically, and what choices are offered to choose from.

In Millionaire-style games the top few (hardest) questions are one's that require incredibly specific knowledge (know all the names of horses who won at least two of three Triple Crown races), have no ability to be logically deduced (all names presented are actual race horse names), and where you can't exclude any of the options (none of the names are the really famous familiar names who have recent movies about them).

A Simplified Proposal To Get Started

Since ultimately only play-testing will really let you figure out what questions themselves are really hard, I will propose a way to get started and get you past the block.

First, write out some questions. Then put them into three piles: potentially easy kid-friendly questions (ones about common pop culture and everyday experience), medium (you don't think everyone will know the answer automatically, but it's not obscure - preferably about a common topic that's covered sometimes in movies and documentaries, like Civil War America or Famous Conquerors), and questions you suspect are going to be pretty rarely known (dig down - if it's about Presidents, don't pick one that shows up on common coins or popular monuments!). Don't worry about strict rank ordering - it's inevitable that some people just know some subjects more than others and they will think one thing is easier/harder than someone else.

Then to produce more fine-grained difficulties in the game, I'd pick "really easy" where you take easy items and pair them so that only one choice immediately appears very plausible. Easy question are then easy items with two plausible choices, one that obviously isn't right, and one more that isn't very plausible but isn't silly. And then you work up to the really hard questions where "you either know it or you don't" and it's very hard to really exclude anything, and it's the hardest (obscure sorts) of questions you manage to have.

Then, because I assume this is computerized: record the time taken to answer the question, "phone a friend" style special bonus invocations (which show the question must have been difficult to the person), and success/failure counts. This will allow you to see whether you were right or not and tweak later - but you can't get there if you don't get started!

So get started! And good luck.

...and that's my final answer.

Answer 2

The "difficulty" of a question is certainly hard to classify. There is a large amount of subjectivity to the issue -- for example, a question about mathematics will likely seem harder to a person with no mathematics background in their education.

You can't classify the "difficultly" of questions in general with an algorithm; you'll have to do it yourself.

One such strategy might be to introduce the concept of topics into your game, and lump each question into one or topics. Within that topic, assign each question a difficulty level (manually, based on your own personal assessment of the question). Then sorting from easy-to-hard becomes a simple matter of sorting on the scale you define for the difficulty (for example, one to ten). The topic does not need to participate in the sort, it's purely for taxonomic purposes.

You'll probably have to tweak those difficulties during play-testing. I'd suggest building in a feedback mechanism into your beta builds -- after every question, show the user the topic and difficulty of the question (as in, "that question had a difficulty of 3/10 in the math topic. Do you agree with that difficulty assessment?") and record their responses so you can make adjustments.

As with all data-collection programs, make sure your beta testers are aware that you'll be collecting their responses to these questions.

Answer 3

The very first thing you need to determine is what is your target audience. Your target audience will help you determine the range of what is considered common versus uncommon type questions. The broader the audience, the more generalized common becomes because the questions will have to span larger age groups of where educational boundaries vary significantly. For example, what kids learn today in school is drastically harder than what our grandparents were taught.

The next thing is to try and categorize your questions based on their commonality. For example, "Who was the first US President?" is a fairly easy and common question that most anyone with any exposure to US history should know. It is far more common than asking "What President gave the Gettysburg Address?".

The next factor to consider is that your questions are being given with multiple choice answers. This obviously can make more difficult answers much easier because of deduction to reason the valid versus invalid choices. This would make questions like my former far easier even for people who may not have had exposure to US History, but also could make the latter question even easier for those whom may not be overly familar depending on your choices. -- listing Lincoln with other names which were not even US Presidents but perhaps Presidents of other countries.

So yes, it can be subjective but there are ways you can rate your questions in a way that can be labeled as easy to difficult but the key factor is your audience.

Answer 4

Actually there's a simple formula for it but its based on how many people on average would get the right answer in a population.

difficulty = correct - wrong

where correct is the number of people that got it right and wrong is the number of poeple that didn't when asked to a large group of people.

The real question is ...

how do you figure out how many people on average would get the answer to a question?

When you figure that out you have your answer.

Typically, the more specific to particular focus / topic a question is the harder it is so if your questions are separated enough you may be able to "figure these numbers out" with some reasoning.

Answer 5

Is your game online? Then you could use a crowdsourcing approach. Just check how many players answer a question correctly and rate the questions accordingly. This can be made completely automatic.

A simple solution would be to have every question start with a score of 0. When a player answers it correctly, you subtract 1 from the score. When a player answers it wrong, you add 1 to the score. A more advanced algorithm could also take player strength into account (when your top player answers a question incorrectly, it means more than when your dumbest player fails).

You can then calculate the percentile each question falls into and confront the player with questions from higher and higher percentiles during their game.

Answer 6

Questions

I agree with the others that this since this is a trivia game, the "heart" of your game is the questions, and this is where a considerable amount of time should be spent. I would recommend starting with a list of metrics on which to "Grade" each question:

• Topic
• Sub Topic
• Geographic Uniqueness
• Percent Correct Female
• Percent Correct Male
• Percent Correct Overall
• Difficulty of the answers
• Difficulty
• etc

The gender metrics may make sense on questions related to topics such as fashion or video games. The point is not to stereotype individuals, to but to present questions that most reasonably match the level of difficulty at the point in the game where that particular player is at that time.

Use whatever you can think of that will be measurable. Then come up with a whole bunch of questions. Get friends, family, coworkers, the Internet, etc to answer the questions and begin assigning values to the metrics. By doing this, you will likely come up with a number of other metrics, and may even scrap some of the existing ones.

When you come up with other metrics then go over the questions again, then again, and again. The questions are the most important aspect of the game. Players will know if they are being fed the same questions over and over.

One of the other answers suggested using the "intelligence" of the answerer to adjust the grade. Intelligence is based on the number of correct questions, or to be more specific, number of correct answers in that topic, since different people may be very good in different subjects and bad in others.

You can even use those metrics on the fly during the game. Like if a player is answering a lot of questions in a particular topic correctly, then you can decrease the likelyhood of receiving that question as they progress in the game.

Answers

For the answer difficulty, if you are doing the "Millionaire" style game, then there will be answers for each question. The answers will matter considerably. The answers should also be graded based on their difficulty when stacked up against a particular question.

A "hard" question can be answered very quickly if the answers are not well designed, and conversely a "simple" question may become very difficult when the answers are very specific.

Specificity matters too. Players can become frustrated in the early questions if the answers boil down to splitting hairs. But in the latter levels, that will be expected.