11

I think anyone who's ever marked an exam with more than one question or section has wondered how to attack it, i.e.

Option 1) Mark all of the questions in a single script before moving onto the next student's script; or

Option 2) Go through each script and mark all the question 1s, then 2, and so on.

Idle rumination gives good points and bad points for each side (and possibly some more approaches), for example 1) means you can get through it faster since you're in 'question 1 mode', then 'question 2 mode', ... (i.e. less time spent switching your brain between different topics), and is likely more fair in the sense that you can consider all the answers to question i at the same time and make sure marks are distributed in a reasonable fashion.

On the other hand, marking a script at a time means you get used to a student's hand-writing, and may perhaps give benefit-of-the-doubt (or not) depending on the understanding that the student has shown previously in the same paper (but hopefully not on the basis of external bias from knowing anything about the student aside from what is on the script).

It would be interesting to see whether one technique or another tends to correlate with higher marks given by a marker, or fewer challenges once the student receives the script back (in cases where that happens), for example.

A brief search via scopus, arXiv and popular general-purpose search engines does not turn up anything particularly relevant. Does anyone here know of any published research on the matter?

Improve this question
3
  • 2
    Related: academia.stackexchange.com/questions/3211/…
    – allicoder
    Oct 17, 2014 at 18:48
  • 5
    As a second-hand anecdote, I believe in the book Thinking Fast and Slow by Daniel Kahneman, he recounts trying to grade one problem at a time vs one student at a time (maybe even multiple assignments of each student at once). What he found was, even when he tried not to, he tended to somehow mark a student more consistently with early work when he did one student at a time. There is a consistancy bias - if a student proves themself smart and capable early in the course, one will naturally grade them higher later because we'll assume they understood even if they didn't explain clearly...
    – BrianH
    Oct 17, 2014 at 19:36
  • 3
    ...but when grading one assignment - or problem - at a time, he found that while his grading was objectively more fair, scores any given student received were much less consistent - sometimes good students did terrible work! By grading one question at a time, it makes it very hard for one - intentionally or accidentally - to favor or disfavor any student. This was not a study, but it's from a smart guy, you see, so sub-rational psychological bias insists you should believe my second-hand anecdote even if it's just an anecdote and argument from authority. :)
    – BrianH
    Oct 17, 2014 at 19:40

1 Answer 1

Reset to default
4
+250

The question we'd like to answer is:

How should exams with multiple essay-type questions be graded?

Note: It isn't possible to answer this definitively due to unavoidably large variations in grader performance and exam structure.

Each course topic, exam, and set of graders is going to be unique and your own experience and judgment are likely to yield better results than any prescription of what you should do. This is a large part of the reason both teaching and assessment are challenging and generally require a lot of experience, testing, and adjustment to get right.

While you're own views should necessarily be more relevant for your teaching situation, here is One perspective to consider:

Ideally, we strive to consistently award the same amount of points for the same level of written work. However, it is likely that our impressions about the abilities of individual students (from lectures, office hours, etc.) actually have a negative impact on the overall consistency and reproducibility of the results when marking written exams.

Whitfield et al. compared oral evaluation results to written exam grades and found that facilitator impressions of student knowledge overestimate the performance of the weakest students.

[Whitfield, 2002] "... there is little overlap in the factors which account for facilitator’s ratings and actual student performance ... Our conclusion from these studies is that an assessment given by a facilitator for student knowledge is not likely to be useful."

The take-away is:

knowing which student's answer you are grading will make the range of scores wider (lower reliability) for the same work, and generally lead to higher grades overall.

For the most reliable grading:

the grader should mark answers non-consecutively and randomly from the entire set of exams in order to:

  1. reduce the chance of identifying the student that provided the answer
  2. reduce the comparative influence from one marked answer to the next

The following articles look promising for further investigation, however, I don't have convenient access to these texts, so they are offered without comment:

"The interrelations of features of questions, mark schemes and examinee responses and their impact upon marker agreement," Assessment in Education: Principles, Policy & Practice, Volume 18, Issue 3, 2011.

"Towards a model of the judgement processes involved in examination marking," Oxford Review of Education, Volume 36, Issue 1, 2010.

"Double-marking students' work," Assessment & Evaluation in Higher Education, Volume 19, Issue 1, 1994.

"Responses to written work," Educational Review, Volume 30, Issue 2, 1978.

"A further study of the reliability of English essays," British Journal of Statistical Psychology, Volume 7, Issue 2, pages 65–74, November 1954.

Related SE questions:

double marking ---- keeping focused ---- avoiding bias

Improve this answer
0

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .