How to choose a good grading curve for yes/no exams?

Question

I teach a certain course where I think the best way to test knowledge is using yes/no questions. However, with a simple yes/no exam, a student can just answer randomly to all questions and get, in expectation, a score of 50%, which is not very fair. I thought of several solutions, but each of them has a disadvantage:

1. Give a negative score to a wrong answer, such that the expected value of a student who answers randomly is 0. However, this rule made some students highly anxious, and I realized that this anxiety could unjustly harm their performance .

2. Ask the students to explain each answer. Award points only to correct answers with correct explanations. The problem is that it takes much more time to write and to grade, negating one of the main advantages of a multi-choice exam.

3. Give a grade of 0 to any student with less than 50% correct answers, and give twice the points for each answer above 50% (e.g, a student with 90% correct answers will receive 2·(90−50) = 80% score). However, I fear this might not be accepted by the university, which encourages additive scoring. E.g, students with 60% correct answers will complain that it is unfair to give them only 20% grade.

What is a good way to write an exam based on yes/no questions?

Answer 1

For this situation, I provide three answer options:

• Yes.
• No.
• I don't know.

The grading rubric is:

• Correct answer: +1
• Incorrect answer: 0
• I don't know: +0.5

That way, if the student really doesn't know, they are guaranteed the expectation of a random guess and you get feedback on holes in student knowledge, which can be used to take remedial action. If your aim is more effective student learning, this can be a good strategy.

I've tried to avoid negative marks for a couple of reasons:

Imagine two candidates who achieve the same final score in a negatively marked exam. One candidate answers only part of the paper, losing a few marks for wrong answers. The other answers almost all of the paper, getting far more correct but also getting several wrong and having marks deducted. Which one knows more?

The second problem is that the use of negative marking extends the theoretical range of marks for the exam. For example, if one mark is awarded for every correct answer and one deducted for each wrong answer, the theoretical range of the exam is - 100 to +100 per cent.

—

From the comments moved to chat: someone asked about a slightly different value for the “I don’t know” option. This is my response:

I tend not to use Yes/No questions, but multiple choice for some assessment. In those cases, I make the "I don't know" option worth 1/(N-1) where N is the number of other options. That way, if there are 3 other options, a random guess would expect to get 1/3 but the I don't know option gains 1/2 --- the expected value if the student can knock out one of the three options..

Answer 2

You are, essentially, evaluating a binary predictor. The usual score is known as accuracy (i.e., probability of giving the correct answer). For a random (coin toss) predictor you will get 50% (if the correct answers are evenly distributed between yes and no).

Note that the condition that the correct answers are distributed %50:%50 between yes and no is itself a huge hint to the students if they know that in advance.

I suggest that you consider two alternative metrics:

Matthews correlation coefficient

Same as Phi coefficient, Pearson correlation coefficient.

1. Range: from -1 to 1
2. All correct: 1
3. All wrong: -1
4. Random: 0

Proficiency

Measures the share of information (percentage of bits) contained in the true answers which is captured by the submitted answers (information here means entropy of the distribution).

1. Range: from 0 to 1
2. All correct: 1
3. All wrong: 1 (sic!)
4. Random: 0

Note that the scores of "all correct" and "all opposite" are the same. This is a feature, not a bug: the person who got all answers wrong has probably confused the labels ("yes" and "no") but knows the subject.

It also gives a lower score to "close to random" results than MCC. IOW, it discriminates good results better.

Caveats

Missing Answers

If an answer is missing, it should be replace with a random yes/no answer.

This is correct statistically, but wrong pedagogically because the honest "I don't know" answer shows second order knowledge that should be encouraged.

Correct answer distribution

The metrics behave the same regardless of the distribution of the correct answers, except when all the correct answers are the same (in which case neither metric is defined). This is addressed by the Bernoulli shuffle (step 2 in the protocol).

These are statistical metrics

These metrics make little sense when there are very few questions. Specifically, a mistake in one question may have a very different cost depending on the question (this can be addressed by repeating the Bernoulli shuffle many times and taking the average score - see protocol step 2 below).

Additivity

These metrics are not additive: you cannot score two halves of the test separately and somehow combine them to get the total test score (let alone add the scores for the two halves).

This can make these metrics a very hard sell to both students and administrators.

Protocol

Given the above, the correct protocol for these metrics is:

1. Prepare hundreds of questions (it is okay for all questions to have the same correct answer).
2. Using a Bernoulli rng with p=1/2, for each question, map yes/no to A/B (i.e., for about 50% questions, yes will be A and for 50% it will be B, and for half of questions the correct answer is A) - this is done completely behind the screen. Neither the students nor the professor need to be aware what the actual map is.
3. Score the resulting binary predictors (now both the true base rate and the predicted base rates are the same - 50%).

The dependency on the Bernoulli rng does introduce some noise (order 1/N where N is the number of test questions). I.e., the standard deviation of mcc and proficiency on a test with 100 questions will be under 1%. Since the steps 2 & 3 are automatic, they can be repeated many times and the mean or median score used.

Answer 3

Sure, a student recording random guesses on a yes/no or true/false exam will have an expected score of 50%. So? Under most grading schemes, this is a solid F.

A more pertinent question to ask is "What is the probability that a randomly guessing student will receive a passing grade / a B or higher / an A?"

For this problem, exam length is your friend! Let's assume that each question is an independent trial and that correct and incorrect answers are equally likely (p=0.5). This places us in the context of the Binomial Distribution. I'll consider a passing grade to be a score of 70% or higher and an A to be a 90% or higher, with a correct answer getting 1pt and an incorrect answer getting 0pts on each question (which meets your institution's preference for additive grading).

• For a short 10 question exam, a random guesser has a 17.2% chance of passing and a 1.07% chance of getting an A.
• For a 20 question exam, a random guesser has a 5.77% chance of passing and a 0.0201% chance of getting an A.
• For a 50 question exam, a random guesser has only a 0.330% chance of passing and a 0.000000210% chance of getting an A.
• For a 100 question exam, a random guesser has only a 0.00393% chance of passing and a 1.52*10^{-15}% chance of getting an A.

While there are many caveats to this rough analysis, this should highlight that random guessing is not a viable strategy for an exam of reasonable length. Keep in mind that the probability of passing is also the same probability that they get less than a 30% on the exam! Along the lines of this statistical analysis, there are many similar takeaways: a student who can make an educated guess (say p=0.7) at each question will almost always outperform a random guesser; a student who knows the answer to the first n-1 questions and then randomly guesses on the last question will do even better. Another takeaway would be that a multiple choice exam has a better likelihood of punishing a random guesser than a yes/no exam.

Personally, I don't find a multiple choice or yes/no exam all that great at evaluating student learning, as it generally hews towards regurgitation of facts and technicalities of wording, as opposed to demonstrating critical analysis and synthesis. If you do persist with grading yes/no or multiple choice, here's a few personal opinions:

• Use a random (or pseudo-random) process to determine the answers. Exams where every correct answer is choice (B) are cute, but are not a good evaluation tool. Students will also pick up on patterns in the order of correct answers, leading to meta-gaming.
• Use a simple metric (0pts incorrect/ 1pt correct or similar) and be sure to explain whether or not they will be penalized for guessing---exams are stressful enough without needing to consider game theory when recording each answer.
• Make the exam sufficiently long. This will help even out the noise from random guessing and make each individual question less stressful. Consider the extreme case of what it would feel like to take an exam with a single T/F question.
• On the flip side, make sure that there is enough time to give a reasoned response to each question.

Edit

I was unaware of the different correlations between raw percentage score and typical grades, as pointed out in the comments. My thanks to @curiousdanii and @cfr for pointing this out to me and my sincere apologies for the ethnocentrism! I greatly appreciate the information from academia.stackexchange that can help us peer beyond our institutional bubbles.

Having only graded within the American system, I don't have a good qualitative feel for the various grades in other systems. However, I feel that a central theme of my answer is still worth considering: random guessing is almost always an inferior strategy to studying and answering questions correctly. As such, I would give serious thought to whether it is truly a problem that you need to control for when designing a rubric.

If pressed, I think the most natural way to account for an unacceptably high expected value (relative to a given grade scheme) would be to switch to a multiple choice format. As with a good T/F question, strive to make all the answers outwardly plausible. Having a "joke" response that can immediately be ruled out accomplishes little.

Answer 4

One possibility that solves all of the listed problems: Grading on a curve.

Granted, Curve Grading is something that gets a lot of peoples' hackles up (either for or against) but this is a situation that it's practically built for: you have a range of results that don't line up with fair letter grades but that are sorted/ranked by degree of knowledge.

Before, a student's 75% would be a 'C', even though they only actually knew the answers to half the questions (and guessed the other half.) On a curve, though, that student's 75% would likely give them a low or failing grade since most other people in the class would hopefully have a higher success rate.

Edit: For clarification, when I say "Grading on a Curve," I mean that X% of the class will get an 'A', Y% will get a 'B', etc. Heck, to be honest, if you explained up front, "I'll be grading this on a curve because, well, if I didn't, people could get 50% for just guessing everything randomly!" and made the curve semi-generous, then you would engender very little ill-will amongst the students.

Answer 5

There are a number of papers published on the over the years. A table of options is shown below from Bandaranayake, et. al., Using Multiple Response True-False Multiple Choice Questions, Royal Australian College of Surgeons, 1999. Some of these consider the construction of "multiple true/false" questions, that is, questions that have one "stem" and several (weakly or strongly) linked true-false statements:

Another option is suggested by Frank Reid, An Alternative Scoring Formula for Multiple-Choice and True False Tests, Journal of Educational Research, 2001:

Some other papers I found include the following:

• Multiple True-False Questions; Hill, G. C.; Woods, G. T. Education in Chemistry, 11, 3, 86-87, May 74
• Scoring Multiple True/False Tests: Some Considerations; Gross, Leon J., Evaluation and the Health Professions, v5 n4 p459-68 Dec 1982
• Burton, Multiple choice and true/false tests: reliability measures and some implications of negative marking, Assessment & Evaluation in Higher Education, 2004.
• Burton R. Misinformation, partial knowledge and guessing in true/false tests. Medical Education [serial online], 2002.
• Tsai F, Suen H. A brief report on a comparison of six scoring methods for multiple true-false items. Educational & Psychological Measurement, 1993.
• Muijtjens, Mameren H, Hoogenboom, Evers, Vleuten C, Muijtjens. The effect of a ‘don't know’ option on test scores: number-right and formula scoring compared. Medical Education, 1999.

Answer 6

Another way to help lower the guessable score while still rewarding the students who have prepared themselves for the exams is to group like questions into one, where answering all the questions about one topic is showing mastery of that topic and earns full credit, whereas only knowing some of the answers is worth less credit.

This is easier to explain with an example. Let's say I'm preparing an exam for a US History class for the unit on the events leading up to the revolutionary war. Some of the events that I might want to be sure my students have studied and understood could be The Establishment of Jamestown, Virginia as the first permanent English settlement, Bacon's Rebellion, The Boston Massacre, The Boston Tea Party, and some others.

When preparing the question on Bacon's Rebellion, I might want to check that the students know when and where it was. So I could set up the question like this:

Bacon's Rebellion (4 points. Possible scores: 4, 1, 0):

• A) Bacon's Rebellion was in 1676. T F
• B) Nathaniel Bacon led the rebellion in Jackson, Mississippi. T F

(Answers: A) True, B) False, it was in Jamestown, Virginia)

If the student gets both questions right, they get all 4 points. If they get only one right, they still get one point. If they get both questions wrong, they get 0 points.

This scoring system would have to be thoroughly explained to the students before hand and written at the top of the test, but to give you an idea of how this distributes the exam scores, if the exam is made up of questions each with two parts like this and a student simply guesses on every question, they have an expected average score of 1.5 points per question, or 37.5% on the whole exam. Yet for a student that knows most of the material and knows 95% of the answers is expected to get about a 92% on the exam.

By awarding one point instead of 0 for the partially correct answer, this helps mitigate the students complaints of "But I knew that the rebellion was in 1676, I just forgot where it was!" because they are still getting partial credit, just not as much as they would get if they showed a mastery of the material by being able to answer both questions.

By never giving any negative points either, this helps with the anxiety you were mentioning your students face with that prospect.

Two tips if you decide to implement this idea:

1. Stick to the same format for the whole exam, and make sure to explain to the students what they are going to see. Either make them all questions with two parts, or do something else, but don't mix them up. Students may become caught up in trying to figure out how the questions are scored, and will lose valuable time that should be used showing their knowledge of the material.
2. Don't try to group more than two questions together in this way. The math might look nice how the expected score of the person guessing drops greatly with each question you add about Bacon's Rebellion, but from the student perspective, if he or she has studied and knows most of the material but just can't remember one little detail that happens to be something you ask, that student immediately loses the majority of the credit for a question that they might know a lot about.

Answer 7

One easy scheme that has been used by some mathematical olympiads is to make every question's answer to be an integer from 1 to 999. If you are careful in setting the question, you usually can make the probability of guessing the correct answer (even after some common sense elimination) to be no bigger than 10%. For example, if the question asks to find the length of some line segment in some geometric construction, engineer the construction so that there are no obvious inequalities that can bound the length to less than 100.

The advantages of this scheme is that the exam is easy to grade and yet difficult for students to get undeserved credit. I am strongly of the opinion that mathematics students should be graded based on proofs, but if you are considering multiple choice questions at least this scheme is a far better option.

Answer 8

a student can just answer randomly to all questions and get, in expectation, a score of 50%, which is not very fair

Not really, unless your grading scheme also gives a high grade to scores above 50%. If you assume the scores follow a normal-like distribution (they almost always will) then the mean will come out somewhere well above 50% and the guessers who got 50% will end up with a pretty negative z-score, and thus a low grade (possibly F, depending on how you set it up).

I think the real issue with true/false questions is that you actually have pretty good odds of winning the lottery, and getting a good grade without knowing the material. Because of this, at least multiple choice or fill in the blank questions are much better. Essay questions, as you mention, are of course the best but much more laborious to actually grade.

Note that you can always convert T/F questions to multiple choice, by using "which of the three above statements are true?". This way it is impossible to guess independently, so the expected score can drop from 50% down to 12% (but in reality will be 20% or 25% because of the number of choices).

Answer 9

After reviewing all of the previously posted answers, I can't see any way in which the inherent problems of a yes-no exam can be overcome. It is simply not a good exam format. You can't extract a useful quantification of your students' knowledge and skill from it.

You can't, but your students can. So if you want to set yes-no exams, do so with the purpose of giving your students a regular purpose for reviewing what they have learnt. Give these exams a nominal 1-2% contribution to the final grade, with a max of 10% for all the yes-no exams combined, and then set one or two more involved assessments to make up the rest of the final grade.

Answer 10

Each yes or no question can be answered in three ways: yes, no, (blank).

State at the start that each correct answer has a +1 score, each blank has a 0 score, and each incorrect answer has, say, a -0.5 score. This discourages guessing but has a lesser penalty than a -1 score. The range of possible grades would then be from -50% to 100%.

Answer 11

You can also make a combination of a linear and a non-linear scales. Say, you give 5 problems with 4 yes/no questions in each one. To determine the problem score from, say, 0 to 20 you can use any reasonable (=monotone and explicable to the students) function on ${0,1}^4$ you wish (you can even assign different values to different questions) but then you just add the scores for the problems. This eliminates the threshold effect that is the main disadvantage of any "cutoff amplification" of just one final score but preserves the general idea that correct random guessing isn't worth much.

As to the common smooth amplification techniques, the ones I've seen in action are $x^2/100$ (so the lucky random guy with nominal 60% gets only 36) and $10\sqrt x$ (if everyone has really low score, this stretches the bottom part and boosts the morale a bit).

Answer 12

If there is a reason why you are considering 50% to be of some benefit, you could restructure the course's grading structure to not be based on the Score (percentage), but instead be based on the Grade (A-F), using the same scale as for GPA calculation (A=4,F=0).

This would mean that scoring 0-59% on any test or assignment would result in 0 points contributing to the final course Grade.

This will make simply guessing on a True/False or Yes/No test to be statistically useless.

I will state that I believe that worrying over a student improperly "gaming" the system to score a failing grade that might be higher than another failing grade he might have received if he had properly taken the test, is stressing yourself out for no useful purpose.

---

The more I think about it, the more I have a problem with using GPA within a class. For one, it would be almost impossible to get an A, as that would be a perfect score.

Perhaps Scoring for assignments could be

• F=0
• D=2
• C=4
• B=6
• A=8

With the final course grade being

• '>=7 gives A
• '>=5 gives B
• '>=3 gives C
• '>=1 gives D
• '<1 gives F

Answer 13

I see three main issues:

1. Someone without any domain knowledge can still get points. You say this is unfair, but unfair to whom? People without any knowledge are getting points, but those with knowledge are getting more points. And the grade thresholds tend to take this into account; the thresholds tend to run from about 50% (F) to 90% (A), so when you take the free points into account, that's effectively 0% knowledge is an F, while 80% knowledge is an A.

2. Negative points make people feel bad. The importance you should give this depends on the level. For elementary school, psychological issues are prominent, but by college, it should be less of an issue (although not entirely eliminated). This can be reduced by presenting students will mathematically equivalent situations with different formulations. For instance, with a 100 point test, you could tell students that they start the test with -50 points, but they get .5 points back every time they skip a question.

3. People who guess do better than those who leave questions they don't know blank. This does create a certain sort of "busy work" for students, having them choose between randomly filling in answers they don't know, or missing out on "free points". Note that this isn't addressed by including a "don't know" option, since they'll still have the busy work of marking "don't know" for all those questions, instead of just skipping them. There is, however, an advantage here where you're basically asking students "Okay, you don't know what the answer is here. But which do you think is more likely?"

There's no way you can fully address all three of these issues, so you're going to have to choose at least one to ignore. You can give half a point to each question left blank, which completely takes care of (3) and mostly takes care of (2) (students aren't getting negative points for getting questions wrong, but they are missing out on the points they could have gotten if they left them blank). You can leave things as they are, which fully addresses (2), but doesn't address (1) or (3). Or you can give negative points, which addresses (1) and (3), but not (2).