25

Our students recently sat an exam and A LOT of them failed. As part of the moderation process it was noted that there were some questions that none of the students had answered correctly. Is there any justification (evidence based if possible) for removing the 'difficult' questions?

4
  • 19
    Your mention of the moderation process suggests this is in a UK setting rather than a North American setting - is that the case? I mention this because there are considerable differences in the "workflow" and "freedom for manoeuvre" between the two systems.
    – Yemon Choi
    Aug 17 at 14:54
  • 4
    Is there a set of written rules or guidelines at your uni?
    – Dace
    Aug 17 at 15:53
  • 11
    Yemon Choi asked a relevant question and I may even go further in stating that there are almost 200 countries, each with its own culture, that are not UK or in north America. Adding a location would help.
    – Johan
    Aug 17 at 22:04
  • 1
    Please don’t write answers in comments. It bypasses our quality measures by not having voting (both up and down) available on comments, as well as having other problems detailed on meta. Comments are for clarifying and improving the question; please don’t use them for other purposes.
    – cag51
    Aug 19 at 20:05

11 Answers 11

37

One worry I have in such situations is that removing some of the questions has a different impact on different students. If there is an issue with a question and some of the students spent an inordinate amount of time on that question then it can affect how they do on others because of the time factor (assuming a timed exam). This impacts the fairness of the solution.

There are several reasons for what you are seeing including ill stated questions and poor instruction on the required material.

If you want to have a completely fair solution, just void the exam itself. Repeat it with different questions after an analysis, perhaps. Another fair solution is to give everyone full marks. If grading has any competitive aspects, such as limits on grading levels, then giving everyone minimal pass isn't a fair solution. Likewise giving full marks on just the difficult questions is also potentially unfair to some students, again assuming that time is a factor.

Note that another possible reason for this is that you just have a statistically unusual set of students. That happens, and, statistically, it must on rare occasions. But most experienced professors will have already recognized that from other indicators.

9
  • 10
    One suggestion -- if you void the exam, if anyone did particularly well on it, you might award them extra credit or find some other way to make their work "count".
    – academic
    Aug 17 at 11:55
  • 8
    Why not lower the threshold to pass (and thus the entire grading scale)? It is what is done at my university. It would impact all students somewhat equally.
    – SirHawrk
    Aug 18 at 5:45
  • 10
    We had such a problem in high school, that a teacher gave a faulty value (show that x=k) but he copied k wrongly. As such he offered anyone who wanted to take a new exam, or to be happy with the result they achieved in the first exam (without the faulty question). I'd wager that this is the fairest solution. If you scored highly on the first one, you do not need to void your good grade. Only caveat was, if you took the 2nd exam, that grade stuck, no matter if it was worse. Aug 18 at 10:00
  • 6
    @infinitezero We had that situation in one of my undergrad classes as well (it is probably quite common unfortunately). There were several errors actually. After a formal complaint by a rather large number of students everyone was offered a new exam, but in this case the highest of the two scores were awarded. That seemed fair to me, as there can be lots of reasons to perform poorly on the second exam (other commitments and work-life balance in particular comes to mind).
    – glaux
    Aug 18 at 12:58
  • 1
    I don't know why you think voiding the exam or giving everyone full marks is fair. It's a uniform solution, but it's not a solution that will seem fair to students who did well/put in a lot of effort.
    – Kimball
    Aug 19 at 0:37
22

I don't have an evidence-based answer, but I've always taken the position that if a preponderance of students miss a question either the question was faulty or my teaching was faulty. Either way, it was my fault, not theirs.

I removed such questions from the computation of the grade and adjusted the point values of the remaining questions, with the caveat that removing question(s) would never lower a student's grade. I computed the grades with the question(s) in place, and again with the question(s) removed. If any grade went down I added an adjustment to that student's grade to return it to the original score.

If I removed a question from any but the final exam, I also cautioned the class, "You will see that question again, so if it needs more explanation, be sure to ask me."

PS: As Buffy has commented, I always warned the class that the questions have equal points, so make two passes through the exam and answer the easy ones first.

1
  • Very commendable approach, this is the right way. Aug 20 at 14:09
17

[UK-focused answer] If the academic marking the exam, the internal moderator, and (when the external examiner visit eventually takes place) the external examiner all agree to it, then yes, it can be done. What standard of justification is required for it is up to those people.

One way of justifying such an action that I've used in the past: if there are some questions on the exam that have already been used in previous years' exams, then you can plot student score on the exam overall against student performance on those questions, both for the current year and for previous years, then try fitting a straight line to each plot. Taking the scores on the repeated questions as a directly-comparable indicator of student ability across year groups, you can say that, if the fits indicate that students of the same ability have typically been getting lower overall exam scores this year than in previous years, then this year's exam is too hard (in a criterion-referenced sense), and this year's exam needs to be adjusted, perhaps by removing the super-hard question.

3
  • 6
    That only works if the students haven't had the opportunity to practice from past papers. If they have, you're measuring a different thing the first and subsequent times.
    – wizzwizz4
    Aug 17 at 23:57
  • 4
    @wizzwizz4 True, but the second and subsequent times remain comparable. Sorry, should have mentioned that. Aug 18 at 8:45
  • 8
    Also worth pointing out from a UK perspective (and the use of the word "moderation" in the question makes me think this is the case), is that the chances of being allowed to offer a new exam are next to 0. Exams need to be taken to happen as specified in the module definition, be in exam weeks, verified by external examiners at the correct time, and results are needed more or less at once for continuation or graduation. Aug 18 at 14:31
10

I think in this case it might be better to adjust the grading curve instead. Suppose your exam has a theoritical maximal score of 100. After grading it you notice a few questions were too hard and these questions are worth 20 points.

The initial plan was to require 90 points for the top grade and 50 points as the minimal passing grade. This would result in a lot more people failing the exam than you intended to fail.

I would suggest to require say 72 points for the top grade and 40 points as the minimal passing grade (adjust as needed). For a student who scored zero on the too hard question this is exactly the same as removing the too hard questions from the grading (that is 90% for top grade and 50% for passing). But if a student still got some points on the too hard questions this method still gives them credit for it whereas removing the questions would not. The students in the exam didn't know these questions were too hard and might have spend a considerable amount of time in the exam trying to solve them. They shouldn't be punished for that.

7
  • 3
    This is not necessarily possible. At my university, to make grading more fair and transparent, the grading scale had to be communicated to the students at the beginning of the course. Changing the top grade from 95% to 70% during the semester would not have been allowed according to uni rules. Aug 17 at 13:55
  • 2
    @Sursula-they- You have to obey your university rules but it seems very strange to me if that would be forbidden but taking out a few questions from the exam after it was written was allowed.
    – quarague
    Aug 17 at 17:07
  • 6
    I've had classes where this was the effective outcome, but with a clever way of communicating it: The instructor turned those questions into "bonus questions", which give points as normal but aren't included in the calculation of grade boundaries. This way students who did all but the too-hard questions can get full marks, but those who solved the hard question didn't get less than before.
    – w123
    Aug 18 at 9:26
  • 2
    @quarague In the UK, grade boundaries are generally set at the level of the university. Aug 18 at 14:33
  • 2
    +1 for curve. The better way to "curve" in this case (in my opinion) would be to make it so the highest score became the benchmark. This is simpler than playing around with thresholds until you get a result you like. If the exam has a theoretical maximum score of 100, but the highest score was 87, everyone would get graded out of 87. Since none of OP's students got all the points, this will increase all of their grades while keeping their relative grades the same. Aug 18 at 19:38
4

I try to not make the exams too hard, but sometimes I don't reach my goal. I've come up with a "curve" policy that I think is fair, and that students haven't complained about.

This is how it reads in my syllabus:

Exams are curved to a B– mean at worst (score mapped to standard normal curve + 2.7), though adjusted such that a numerical grade of 80 on an exam is never lower than a 3.0. The instructor will adjust this as necessary to reward good performance. For example, an exam in which every student scores 95 or higher would result in A’s for the whole class on that exam.

UPDATE: To expand on the question's premise, I would do everything I could to not remove a question from the exam post-hoc. I think there can be tremendous fairness issues over that. Think of a student who spent an inordinate amount of time on the question, to the point that other questions get ignored, only to find that the question didn't count, vs a student who didn't think they'd be able to answer the question and skipped it entirely, comfortably finishing the rest of the exam. I can say that I believe the first student is harmed whether you leave the question in or not, but removing the question post hoc certainly doesn't help. It's a bell you can't unring.

3

My solution in a similar situation may not be applicable or fair or legal in yours, but here it is.

Although I try not to, I often write exams that are too hard (in part). I want them to be learning experiences for the student, not just evaluations.

By the time I give exams in a course, I know from other interactions that there are some students on track for an A and some on track to fail. With that information I can adjust the grading curve. The top scores, no matter how low, represent outstanding work.

1
  • 3
    My practice was somewhat similar. I warned students that there would be on very hard question on the exam. Students are also often warned to write answers for the easiest (to them) questions before tackling the more challenging ones.
    – Buffy
    Aug 17 at 14:48
3

I have been in a similar situation. While a lot of this will depend on the level of your students (i.e., I would treat freshmen differently from graduate students), my process was to look over the test carefully and re-evaluate how I assembled it. Were the questions fair (hard is OK)? Did the students have a chance to work similar problems as homework or during class?

Then, I'd look at the overall distribution of grades. Did some students ace the test (or come close)? If so, the test was fair. Was the distribution "double-humped" (indicating two populations of students - those who studied and those who did not)?

Testing should, in my opinion, be fair. If it was your fault (e.g., the test was too long or too difficult), then I would remove the "offending" questions for the next exam and I would curve the graded exam appropriately. If it was not your fault (i.e., the students just didn't study), then I'd leave it as is. Those are, of course, the two extremes; you may need to combine those (with potentially other) solutions to meet your needs.

3

I wouldn't remove the question altogether, because that might feel too unfair. Instead I would propose a system which rewards those who did well on the question, but does so without punishing the majority who did not get that tough question.

In the past, I had a 4 question exam to grade, and one of the questions had an abysmal median over each of its parts. That 'hard' question was worth 30 points, with the other 70 points spread over the 3 remaining questions. So, I made the exam worth 70 points, and gave students a score based on their 3 best questions. For most, this meant the hard question was dropped entirely. If someone got the full 30 points on the hard question, those 30 points would replace their lowest question score, which presumably would be a question worth 15 points. This means that they would get 15 bonus points, or just over 120% on the exam.

This scoring system received no complaints from students, as it could only ever help them (either dropping their bad scores, or giving them extra credit).

2
  • PS just in case it's not clear, even though the exam has its point value changed, you would still keep it worth the same weight in the final grade calculation
    – Taw
    Aug 18 at 14:20
  • This answer works if the numeric score is the only thing that matters from the students' perspective. But if their rank compared to other students is important, this system will hurt some. In particular, a student who follows the common exam advice of "allocate time to each question in proportion to its marks" will be disadvantaged against one who chooses to neglect one question entirely and focus on the other three. Aug 20 at 3:35
2

My June 1980 O Level Mathematics exam (UK) was voided after 86% of us failed it.

Several of the questions related to things none of us had even been taught (our tutor told us he thought they'd mixed up the O level paper with the A/S paper). The A/S paper was the halfway point between O level and A level.

We all got a resit in the August.

1

Your school might have a policy. Check that, because we can't know. There may be laws that apply as well, depending on location and the level of your institution. If you have a student union, council, representatives, etc., consider seeking advice from them as well as other faculty.

You have several good answers already (though I disagree with grading curves on principle) but I'd like to focus on a question/challenge that's not getting as much attention.

Student success rate can be an indicator of exam quality or "appropriateness", but because of confounding factors you should not let it be the ultimate judge. As an educator you are often in a conflict of interest between having your students pass (keeping them and your institution, at least temporarily, happy with you) and having your students learn (doing your job).

"Most students failed" - or even "all students failed this question" - is not enough evidence to conclude that the exam - or question - was too hard. It may be that students were distracted by other events, that you had a statistically anomalous (but still likely to happen once in a while) group of students, or that they weren't taught the concepts well. Adjusting the exam won't give them the knowledge they are expected to have.

Therefore the next step should be to determine whether the questions were reasonable, not based on how many students answered correctly, but how many students should be able to answer correctly. Daniel's answer explains a method for just that.

If the question is reasonable, you probably want to keep it and address the failures some other way. Perhaps remedial classes - with a different approach or instructor if possible - and a second attempt at the exam. We have done evening classes to help classes catch up in relatively short order, but of course that's not always possible. If this was not the final exam of the class, you can be more charitable with grading it, but make sure to use similar questions later on.

If the question was not reasonable, you should remove it from grading or find another way to compensate students, and consider offering a second attempt to any students who feel that they spent too much time on the difficult question. Again, other answers handle this case already.

1

I can't speak for the OP's situation, but how about offering the missed questions as a homework set, or even the entire exam for that matter, and giving half the missed credit back? That way students who worked hard on the hard problems will have a leg up on those who didn't. It's also a great way to force students to review what they should have known for the exam, so it reinforces actual learning instead of cram-and-forget.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .