What to do when students answered more questions than instructed during an exam?

Question

I am a 5th year grad student. This is the first time I am allowed to teach a higher level major course on my own. We just had our midterm today and I am confused as to how to proceed.

On the midterm that I created, the instructions were to "attempt only 5 problems" out of a total of 8. But I didn't specify on the exam what would happen if the students were to attempt more than 5 problems.

I noticed half of the class (11 of 20) did follow the instructions and only handed in 5 problems or clearly marked which one are to be graded. But there are those who did more than 5 and did not specify which ones I should grade, perhaps hoping that I would pick their highest 5 problems or they simplify forgot.

What should I do in a situation like this? I know it is "standard practice" to simply grade the first 5 problems in numerical order that they have attempted. Unfortunately it seems most if not all students have no clue how to do problem #2 and #3 (these were fair questions, but required students to be clever), therefore if I were to follow this standard procedure, the students who attempted more than 5 (which basically means they attempted all) will almost certainly get a failing grade, which seems a little cruel.

(I listed on the syllabus "rough" grade cutoffs, i.e. A = 90+, B = 80+, C = 70+, etc. with pluses and minuses. The grades of students in the class are basically clustered at the moment, with a bunch of students around 90-95%, then a couple around 70-80% and then some around 30-50%, with not many outliers.)

Any advice would be appreciated!

EDIT: I asked two faculty members they both told me to simply grade the first 5 attempted questions in numerical order, and the difficulty of each question shouldn't matter. They said the same policy is used on qualifying exams. Sorry that I was just overthinking. That's what I'm going to do.

Answer 1

My view is that we should make sure grades are good predictors of underlying ability. This answer reflects that. Perhaps it slightly overestimates a small number of students, but that is much better than vastly understimating/failing them (in my view, capriciously).

Option 1

Since the exam just happened, and with so few (~10) students, I would simply email them and state something like:

It's unclear to me which five questions you would like graded, please let me know by [some short deadline] or I will grade #1-5.

The short deadline - perhaps by the end of the next business day - is so they don't have too much time to game this system. In my view, if they are extremely likely to drop #2 and #3, any possible advantage they may have gained by discussing their answers and realizing which ones they did really wrong is well ameliorated by having them still have to pick five out of the remaining six and already having lost time working on more than five problems.

Option 2: Just grade the five most complete answers. This is probably what they want and what they meant to do.

If this still feels unfair to you after getting their preferred five questions, you could grade all students on their best four out of five. (This is what I'd do if this happened to me).

Finally, you could pick randomly out of the problems they started, but given these are upper-division courses, I feel giving them a little bit more benefit of the doubt feels more reasonable.

Answer 2

Mark the five best answers for each student. Presumably the fact that they completed more work than necessary means they spent less time on those answers: hence, they have already 'penalised' themselves for not following instructions.

Next time, make sure the instructions are clear on what happens if they don't follow instructions.

Answer 3

I would like to confirm the answer from Avid, with following thought experiment:

A student gets the eight questions and starts solving them, in the sequence from easiest to most difficult.
Once the student has finished five questions, he realises he still has some time left, and decides to solve the sixth one, and the seventh one, and finally the eighth one, which is the most difficult.

As such, the student has proven:

• To be capable of solving all problems, even the most difficult ones.
• To have an ambitious work ethics.

I could not disagree more with the reaction of Cag51, as if the "skill" to follow instructions is more valuable than the proof of higher competence, especially in a school environment.

Answer 4

My most important advice is not to do that again. And, make sure students read and understand the rules. Not all will find them obvious, though with 20 it is easier.

There is no perfect solution, but there doesn't need to be. If you use coarse graduations, then it probably doesn't matter if you take simple actions. And if you don't grade competitively in any way then you can assure that it doesn't really matter at all. If your attitude is "I'm here to give points", rather than "I'm here to withhold points" then you have a solution. But since it is on you, then it means more work for you (another reason to avoid a repeat).

Do something like this:

Peruse the other students first, to get a sense of what is known and what is not - generally. Then look at the students who answered more than needed without indicating which were to be included and see if their knowledge generally matches that of the others. Look at everything they did, without throwing out questions arbitrarily. You can, judiciously sort them into the coarse categories without applying numbers.

It may turn out that the few students have really demonstrated mastery overall, so should get high marks. They may have done poorly on most questions - low marks. But if the graduations are sufficiently coarse (as is normally true in the US at the end of a course) then the sorting problem isn't as hard as you imagine.

But a distinction between 89 and 91 is probably (definitely) beyond the bounds of the possible.

And note that the Sorting Hat at Hogwarts only had to distinguish four categories.

Remember that you are grading individuals, not the class as a whole. Each student demonstrates a level of mastery or fails to do so. All you need is a rough measure, mixed with a bit of generosity. This works as long as being generous is applied generally over the course so that individuals aren't treated unfairly by the grading system itself.

And the big big lesson is that you are an educator, not a grader.

You'll get better at this, I suspect, but don't repeat old mistakes.

Answer 5

While I'd broadly agree with the advice in your edit, I would mark the first 5 in the answer booklet, rather than questions 1-5.

Answer 6

When this happens you generally have two possible options:

• Grade the first five problems.
• Grade all of them, and then take the final score as the sum of the best five problems.

It's obvious that the second option is better for the student, while the first is better for you (since it takes less work). Since the instructions clearly said "attempt only five problems", you should take the option that is better for you. You can't control what students do in exams, but you can stop them from abusing you as a grader.

Frankly I'm surprised this hasn't already been drilled into your students by their high school teachers. If you do take the second option, I would emphasize afterwards that it's a one-off act of mercy, and not to expect similar favours in the future.

Answer 7

First, be transparent about what happened and how you are addressing the situation. Document your grading methodology and share enough details with the class so that no one will have reason to feel aggrieved.

Second, your goal is to maximize the fairness in your treatment of all students. Unfortunately your unclear instructions can leave some theoretical possibility for someone to feel (or claim to feel) like they are being treated unfairly no matter what you decide to do. But the scope of this theoretical unfairness is quite small, so if you proceed with common sense and good faith, I think you (and everyone else) will be fine.

Now, as for what to do, I think the following is a reasonable option:

1. For each exam, grade all the questions answered by the student. Use their top 5 question scores and as the basis for determining a tentative grade.

2. Review the grade distributions of the two groups of students (those who answered exactly 5 questions, and those who answered more). Do you see a sizable difference in the average scores across the two groups? I.e., something that gives you reason to believe the group of students who answered more questions than they were supposed to are enjoying an unfair advantage in the grading?

3. If there is a sizable difference favoring the “rule-breaking” students, consider artificially raising the scores of the second group to compensate for that difference.

4. If there isn’t a sizable difference between the two groups, or there is a difference but it’s pointing in the other direction (the rule-abiding students being the ones who enjoy the advantage), consider not making any adjustments and declaring the tentative grades you calculated as the final grades.

5. As an intermediate solution, if there is some difference but you are not sure how relevant or significant it is, consider using the tentative grades as final grades, but announcing to the class that you are reserving the right to increase the grades of some or all of the students from the rule-abiding group later in the semester, if you feel that that makes sense. That leaves you an opening to be generous in some way that’s entirely at your discretion, for example if at the end of the semester there is a rule-abiding student whose score is very near one of the grade cutoff points and could benefit from a small extra push.

6. Any of the above options can be considered based on not just the statistical calculation I mentioned but also other information that you have that affects your perception about the extent of the unfairness.

If you follow these steps, I think the students will appreciate that there was a minor snafu in the exam instructions and you handled it reasonably and competently. Again, the key is: transparency, acting in good faith, and taking care that any grade adjustments you make are only in the direction of increasing the grades of some students, not in the direction of penalizing anyone.

Answer 8

Novel answer:

Grade all questions answered and then normalize it to out of five.

If all questions are worth the same, then it is simply score * 5/N where N = number of questions answered. Only do this if N > 5.

Also count a question that clearly abandoned early on as not answered- just ignore it.

Pros A student who does equally well on all questions will get the grade the would have gotten answering only five.

Although possible to get a better grade than they would have, it is unlikely. It would require that the student had no insight into which they did better.

Some or many will get a lower grade than they would have but it will not be as bad as just doing the first five- taking as true the OP statement about questions 2 and 3. It will soften the blow of those questions.

So bottom line is there is a negative effect for most who did not follow instructions, but it is smoothed out vs. just picking five to grade.

Cons

It is more work from you.

You will get complaints that you should have taken the top five. And some edge complaints about incomplete answers that the student will say was clearly abandoned.

Answer 9

I find it highly unlikely that half your upper-level class failed to follow instructions unless we are not being told the whole story. Half the class doesn't do something like this unless something was at least somewhat unclear.

Did you announce your exam structure in the lecture before the exam? Is it described in the syllabus? Did you explicitly tell the students to mark the questions they want to be graded or rip out the pgs from the exam book they didn't want graded? Did you announce, verbally at the start of the exam, "remember attempt only 5 of 8 questions", or is it just in the written exam directions?

Written instructions are like the fine print of a 30 pg legal mumbo jumbo contract. No one reads them. They are taking a timed exam! Reading written instructions is a waste of time on basically any other exam. If you surprise your students with unusual rules in written instructions, you are effectively tricking some of your students. The only exception to this is if you allocate a minute or two at the start of the exam explicitly for reading instructions. "Now you will read the instructions on the first pg and not open the exam. Be sure to read the instructions carefully right now."

Furthermore, did you define what an "attempt" is in your written instructions? Suppose you got the exam

1. several well-thought-out and presented paragraphs/calculations etc.
2. a few scribbles or sloppy scratch work
3. a few scribbles or sloppy scratch work
4. several well-thought-out and presented paragraphs/calculations etc.
5. several well-thought-out and presented paragraphs/calculations etc.
6. several well-thought-out and presented paragraphs/calculations etc.
7. several well-thought-out and presented paragraphs/calculations etc.
8. several well-thought-out and presented paragraphs/calculations etc.

I assume under your "grade the first 5 attempted problems" you'd grade 1,2,5,6,7? Correct? If you graded 2 and 3, and I was your student, and it meant I failed the exam, I'd try my best to get you in serious trouble with your University. I'd drown you in paperwork and proceedings for the next year. I'd rally half the class to complain with me. Don't be fooled, most Universities in the USA usually side with the students in borderline cases like this. The students are the customer and you screwed up by not specifying that there is a penalty for attempting more than 5 problems. I highly doubt that half the class was trying to get an advantage. It's much more likely that the students simply didn't understand, didn't know, or forgot your unusual exam specifications. Do you really want to fail a student who knows the material well because of this?

Answer 10

Rule nr. 1 for teachers is that students are always right in a collective sense. A single student can be wrong, but if there are more than a few who did something unexpected, then you are the one who is wrong, the students are right, however inconvenient that is for you.

In this case, this means that you should look at all the answered questions and grade the 5 best, no matter what the rules say. Now, one could argue that the students did in fact make a mistake, but that's a totally mistaken view to take.

The students are not at university to pass exams, they are there to learn and master particular subjects. They are paying for the services offered by the university, including the opportunity to sit exams to measure how well they've mastered the subjects they are studying.

If you are going to grade the first 5 questions leading to students who would otherwise have scored 100% to now fail the exam, you are not delivering a service to the students that they paid for. This is then grounds for appeal, which could even go to a court should the university uphold the decision to fail he students. With many thousands of dollars paid for tuition, no judge would rule that it would have been too much effort to correctly grade the students that would have avoided this absurd outcome.

Answer 11

You assert this is your first time teaching at this level. I noted carefully your statement (emphasis added)

Unfortunately it seems most if not all students have no clue how to do problem #2 and #3 (these were fair questions, but required students to be clever)

Since "most if not all" of the students did not successfully respond to the problems (honestly, did any succeed? your statement is vague on this point), it appears that you yourself have failed to either adequately teach the principles covered by the problems or in your expectations for the students to complete them. This calls their fairness into question. In every instance I have observed of this, the instructor owned their failing and eliminated such problem(s) from having a negative consequence on students' grades.

I appreciate that "clever" solutions are encouraged but when it requires such cleverness that the vast majority are likely to fail (per your own admission), the most ethical response is to remove said questions from consideration.

i.e. only grade 1,4,5,6,7. Students who did not complete the requisite 5 after eliminating 2 and 3 should be graded fairly on their attempts.

Answer 12

I ended up grading the first 5 attempted questions for each student i.e. if a student attempted #1,2,4,5,6,7 then I graded #1,2,4,5,6. For questions that require short responses almost anything other than a blank counted as an attempt. For questions that require long essay-like responses (proofs), anything that stated an assumption and claim counted.

I don't think it's reasonable to say students "forgot" to indicate which problems are up for grading. I think the students who submitted more than 5 questions for grading without indicating which ones they want me to grade, are simply trying to game the system or get an advantage over other students, and here's my reasoning:

1. I have repeated in class instructing them to indicate which questions for me to grade. In the exam students were given a "question sheet" which listed all the questions and were provided "unlimited" number of scratch paper, in which they will put their responses on. On top of that, I have verbally told the class in the beginning of the exam "to only submit or indicate 5 questions up for grading, not more," and also at the end of the exam to "make sure you choose which 5 are to be graded."
2. Some questions require only a very short response which could be totally correct or incorrect (for instance, find the kernal of some map, or, list all the subgroups of some group). I think some students aren't sure if they got these questions correct and don't want to risk losing points for the entire question(s), so they simply didn't choose which ones are to be graded, versus the rule-abiding students who had to choose whether to discard these "questionable" answers.

Thanks for all your responses, I didn't expect this many answers, when I made the decision to grade as I did I only had 3-4 responses.

Answer 13

Your institution should have a rule for how to grade exams in this circumstance. If so, find out what the rule is, and follow it. (It appears from the update to the question that this is what you did and it resolved the issue.)

If the institution's rule is one you disagree with, follow it anyway; it would be unfair to apply a different rule, because students who took the exam may have known the rule, and may have submitted answers according to their understanding that the rule would be followed.

---

If your institution doesn't have a rule to deal with this happening, I personally believe the fairest and most sensible way to deal with it is to mark every answer and then take the best 5. The reason I think this is the fairest rule is because it doesn't penalise a student for starting a question, realising they can't answer it, and then choosing a different question. If a student wrote one or two lines on some question and then gave up, it would be absurd to treat that as one of their five questions when they have written 5 full answers to other questions. That student could lose up to 20% of their mark just because they put pen to paper before realising that they could do better on a different question. (Or if they were supposed to cross it out to indicate that they didn't want it to be marked, then they could lose up to 20% of their mark simply for not putting a line through something they thought would be ignored anyway.)

There are some other answerers and commenters who believe taking the best 5 answers would be unfair on students who only attempted 5 questions. I disagree; suppose the exam is 80 minutes long. Then if a student "followed the rules", then their mark is based on work which they had 80 minutes to do. A student who answers all 8 questions will receive a mark based on work they only had ~50 minutes to do. My own experience (from marking exams where the best k answers count, and students are told so) is that students who attempt too many questions tend to have rushed and made mistakes in most of their answers.

Also, if your institution doesn't have a rule to deal with this happening, then make sure that someone higher-up knows that they need to make one so this won't be a problem in the future.

Answer 14

I have just been in this situation. I solved it by grading all of the problems and calculating the expected value of randomly selecting five problems. In my case, a complaining student actually admitted that he was trying to game the system.

I give choice because sometimes people have difficulties understanding a problem as posed and because sometimes people draw a blank on a particular problem.

Answer 15

Picking the worst answers is the "standard practice" where i live and most fair to the students that did follow the rules. Picking 5 of the best answers is giving an advantage to the people who didn't read the rules. I think what the student has proven by solving multiple questions is irrelevant. Rules are rules and who is to say that people who only solved 5 questions couldn't solve all 8? It is after all, the smart thing to do.

The other option is picking random questions, but that doesn't feel very professional to me.

Answer 16

Simple. Given there are 40 students who answered more questions than allowed, generate 40 arrays with the numbers 1-8 in a random order. In Python,

from random import shuffle
a=[1,2,3,4,5,6,7,8]
for s in range(40):
    shuffle(a)
    print(s,a)

This will give something like

0 [4, 2, 5, 7, 6, 3, 1, 8]
1 [4, 7, 2, 3, 5, 8, 1, 6]
2 [5, 1, 6, 7, 2, 8, 3, 4]
3 [6, 1, 3, 8, 4, 5, 2, 7]
...

Then you only consider the first 5 questions numbers that the student answered. For example, if student 0 answered questions 1 3 4 5 6 8 then you would only consider questions 4 5 6 3 1.

This is very fair because the questions considered are randomly selected. So there is no gaming the system.