Can exam questions be removed for being too difficult after students have sat the exam?

Question

Our students recently sat an exam and A LOT of them failed. As part of the moderation process it was noted that there were some questions that none of the students had answered correctly. Is there any justification (evidence based if possible) for removing the 'difficult' questions?

Answer 1

One worry I have in such situations is that removing some of the questions has a different impact on different students. If there is an issue with a question and some of the students spent an inordinate amount of time on that question then it can affect how they do on others because of the time factor (assuming a timed exam). This impacts the fairness of the solution.

There are several reasons for what you are seeing including ill stated questions and poor instruction on the required material.

If you want to have a completely fair solution, just void the exam itself. Repeat it with different questions after an analysis, perhaps. Another fair solution is to give everyone full marks. If grading has any competitive aspects, such as limits on grading levels, then giving everyone minimal pass isn't a fair solution. Likewise giving full marks on just the difficult questions is also potentially unfair to some students, again assuming that time is a factor.

Note that another possible reason for this is that you just have a statistically unusual set of students. That happens, and, statistically, it must on rare occasions. But most experienced professors will have already recognized that from other indicators.

Answer 2

I don't have an evidence-based answer, but I've always taken the position that if a preponderance of students miss a question either the question was faulty or my teaching was faulty. Either way, it was my fault, not theirs.

I removed such questions from the computation of the grade and adjusted the point values of the remaining questions, with the caveat that removing question(s) would never lower a student's grade. I computed the grades with the question(s) in place, and again with the question(s) removed. If any grade went down I added an adjustment to that student's grade to return it to the original score.

If I removed a question from any but the final exam, I also cautioned the class, "You will see that question again, so if it needs more explanation, be sure to ask me."

PS: As Buffy has commented, I always warned the class that the questions have equal points, so make two passes through the exam and answer the easy ones first.

Answer 3

[UK-focused answer] If the academic marking the exam, the internal moderator, and (when the external examiner visit eventually takes place) the external examiner all agree to it, then yes, it can be done. What standard of justification is required for it is up to those people.

One way of justifying such an action that I've used in the past: if there are some questions on the exam that have already been used in previous years' exams, then you can plot student score on the exam overall against student performance on those questions, both for the current year and for previous years, then try fitting a straight line to each plot. Taking the scores on the repeated questions as a directly-comparable indicator of student ability across year groups, you can say that, if the fits indicate that students of the same ability have typically been getting lower overall exam scores this year than in previous years, then this year's exam is too hard (in a criterion-referenced sense), and this year's exam needs to be adjusted, perhaps by removing the super-hard question.

Answer 4

I think in this case it might be better to adjust the grading curve instead. Suppose your exam has a theoritical maximal score of 100. After grading it you notice a few questions were too hard and these questions are worth 20 points.

The initial plan was to require 90 points for the top grade and 50 points as the minimal passing grade. This would result in a lot more people failing the exam than you intended to fail.

I would suggest to require say 72 points for the top grade and 40 points as the minimal passing grade (adjust as needed). For a student who scored zero on the too hard question this is exactly the same as removing the too hard questions from the grading (that is 90% for top grade and 50% for passing). But if a student still got some points on the too hard questions this method still gives them credit for it whereas removing the questions would not. The students in the exam didn't know these questions were too hard and might have spend a considerable amount of time in the exam trying to solve them. They shouldn't be punished for that.

Answer 5

I try to not make the exams too hard, but sometimes I don't reach my goal. I've come up with a "curve" policy that I think is fair, and that students haven't complained about.

This is how it reads in my syllabus:

Exams are curved to a B– mean at worst (score mapped to standard normal curve + 2.7), though adjusted such that a numerical grade of 80 on an exam is never lower than a 3.0. The instructor will adjust this as necessary to reward good performance. For example, an exam in which every student scores 95 or higher would result in A’s for the whole class on that exam.

UPDATE: To expand on the question's premise, I would do everything I could to not remove a question from the exam post-hoc. I think there can be tremendous fairness issues over that. Think of a student who spent an inordinate amount of time on the question, to the point that other questions get ignored, only to find that the question didn't count, vs a student who didn't think they'd be able to answer the question and skipped it entirely, comfortably finishing the rest of the exam. I can say that I believe the first student is harmed whether you leave the question in or not, but removing the question post hoc certainly doesn't help. It's a bell you can't unring.

Answer 6

My solution in a similar situation may not be applicable or fair or legal in yours, but here it is.

Although I try not to, I often write exams that are too hard (in part). I want them to be learning experiences for the student, not just evaluations.

By the time I give exams in a course, I know from other interactions that there are some students on track for an A and some on track to fail. With that information I can adjust the grading curve. The top scores, no matter how low, represent outstanding work.

Answer 7

I have been in a similar situation. While a lot of this will depend on the level of your students (i.e., I would treat freshmen differently from graduate students), my process was to look over the test carefully and re-evaluate how I assembled it. Were the questions fair (hard is OK)? Did the students have a chance to work similar problems as homework or during class?

Then, I'd look at the overall distribution of grades. Did some students ace the test (or come close)? If so, the test was fair. Was the distribution "double-humped" (indicating two populations of students - those who studied and those who did not)?

Testing should, in my opinion, be fair. If it was your fault (e.g., the test was too long or too difficult), then I would remove the "offending" questions for the next exam and I would curve the graded exam appropriately. If it was not your fault (i.e., the students just didn't study), then I'd leave it as is. Those are, of course, the two extremes; you may need to combine those (with potentially other) solutions to meet your needs.

Answer 8

I wouldn't remove the question altogether, because that might feel too unfair. Instead I would propose a system which rewards those who did well on the question, but does so without punishing the majority who did not get that tough question.

In the past, I had a 4 question exam to grade, and one of the questions had an abysmal median over each of its parts. That 'hard' question was worth 30 points, with the other 70 points spread over the 3 remaining questions. So, I made the exam worth 70 points, and gave students a score based on their 3 best questions. For most, this meant the hard question was dropped entirely. If someone got the full 30 points on the hard question, those 30 points would replace their lowest question score, which presumably would be a question worth 15 points. This means that they would get 15 bonus points, or just over 120% on the exam.

This scoring system received no complaints from students, as it could only ever help them (either dropping their bad scores, or giving them extra credit).

Answer 9

My June 1980 O Level Mathematics exam (UK) was voided after 86% of us failed it.

Several of the questions related to things none of us had even been taught (our tutor told us he thought they'd mixed up the O level paper with the A/S paper). The A/S paper was the halfway point between O level and A level.

We all got a resit in the August.

Answer 10

Your school might have a policy. Check that, because we can't know. There may be laws that apply as well, depending on location and the level of your institution. If you have a student union, council, representatives, etc., consider seeking advice from them as well as other faculty.

You have several good answers already (though I disagree with grading curves on principle) but I'd like to focus on a question/challenge that's not getting as much attention.

Student success rate can be an indicator of exam quality or "appropriateness", but because of confounding factors you should not let it be the ultimate judge. As an educator you are often in a conflict of interest between having your students pass (keeping them and your institution, at least temporarily, happy with you) and having your students learn (doing your job).

"Most students failed" - or even "all students failed this question" - is not enough evidence to conclude that the exam - or question - was too hard. It may be that students were distracted by other events, that you had a statistically anomalous (but still likely to happen once in a while) group of students, or that they weren't taught the concepts well. Adjusting the exam won't give them the knowledge they are expected to have.

Therefore the next step should be to determine whether the questions were reasonable, not based on how many students answered correctly, but how many students should be able to answer correctly. Daniel's answer explains a method for just that.

If the question is reasonable, you probably want to keep it and address the failures some other way. Perhaps remedial classes - with a different approach or instructor if possible - and a second attempt at the exam. We have done evening classes to help classes catch up in relatively short order, but of course that's not always possible. If this was not the final exam of the class, you can be more charitable with grading it, but make sure to use similar questions later on.

If the question was not reasonable, you should remove it from grading or find another way to compensate students, and consider offering a second attempt to any students who feel that they spent too much time on the difficult question. Again, other answers handle this case already.

Answer 11

I can't speak for the OP's situation, but how about offering the missed questions as a homework set, or even the entire exam for that matter, and giving half the missed credit back? That way students who worked hard on the hard problems will have a leg up on those who didn't. It's also a great way to force students to review what they should have known for the exam, so it reinforces actual learning instead of cram-and-forget.