Normalizing grades across multiple graders

Question

We teach a large graduate class with many teaching assistants (TAs), who are involved in grading 4-5 assessments (each contains a large amount of code, and a 4-page report corresponding to the solution).

Each time we grade, it takes an extremely long time-- we first carefully create a rubric, based upon which the TAs evaluate each others' solutions; next we review these grades and come to a consensus on individual line items in the rubric. We also blindly grade a single student submission at a time and re-calibrate the rubric and it's application till we have consensus.

Despite doing all of this, we end up having non-trivial variations in the mean, median, min and max grades assigned by each TA. We have hundreds of students and dozens of TAs, and it isn't feasible to have multiple grades per student submission.

Last Spring, this led to a significant number of grading iterations for each assignment, and we eventually ended up averaging across iterations, rather then converging to a common grade for each student.

We're trying to automate the monitoring and evaluation of grading tasks on our grading portal so that outliers/problem cases can be reassigned, and the lead instructor can identify if a TA is not grading well. Is there any research or anecdotal experience with such grading issues in classes that you have taught/assisted in (and any useful countermeasures)? If not, is there a systematic way where grades can be curved based on the grader, in a way that is fair?

Answer 1

I am not entirely sure if this is an "answer" or a "comment" but I'll supply it as an answer.

First off, I'm answering from the perspective of academia -- not from the perspective of perfect statistical analysis or experiment design or anything like that. So put away any screams that this fails some sort of T test or does not produce a normal distribution -- those aren't the goals of academic work in general.

I think you might be trying to solve this problem the wrong way in all honesty. You state:

We also blindly grade a single student submission at a time and re-calibrate the rubric and it's application till we have consensus.

and then

Last Spring, this led to a significant number of grading iterations for each assignment, and we eventually ended up averaging across iterations, rather then converging to a common grade for each student.

This sounds like an immense amount of work to give students grades. But worse than that, it's clear the earlier work doesn't do anything for you since you had to take the consensus-achieved "re-calibrated rubric" and then if I'm reading correctly redo and jig together a bunch of things to produce grades.

If it seems like an immense struggle to come up with grades, then at least on of the following seems true to me:

1. You and your team are suffering from a strong case of OCD.
2. You're probably not achieving real consensus on how to grade things so much as minimal acquiescence on a single answer.
3. You and your team are fighting against gravity -- i.e. you are trying to counteract a feature of human grading that you should just accept and work around.
4. A misundertanding of the nature of academic work and grading (for more on that topic see this question and especially this answer: https://academia.stackexchange.com/a/31526/20058).

I see three potential solutions:

Rather than try to QC to perfect grade normalization, accept that individual grading differences are in-eradicable and make it so students' work is graded by a distribution of graders and that grades are not so subject to this fluctuation as to be questionable. For instance, have each grader look at a sample of 5 "standards" (an A, B, C, D , F) and see what grades they assign to them. Use this to categorize graders as severe, neutral, and soft and make it so everyone gets a fair mix of graders.

And/or make the rubric explicitly clear to the point where individual differences don't matter. i.e.,

"one point for a program that complies, one point for a program that
executes without crashing, one point for a program that produces the correct output, two points for using a recursive function, one point for mentioning "iterative sort" / "iterative sorting" in the description.

And/or figure out where the individual grading differences happen and minimize the importance of these for actual grades.

Answer 2

I agree with the previous answer in that you do seem to be making more work out of this than is necessary.

For the programming part of the assignment I would suggest defining very specific marking criteria - if the program successfully does this then you get this mark. This should be easy to grade and could even be automated. Some aspects of programing style also could be marked with precise criteria.

For more subjective aspects - the report or perhaps some aspects of code style you probably don't want to be quite as specific. You are going to get variation between markers no matter what you do. However I would suggest that going around iterations as you are saying that you do is more likely to lower confidence in grading than having a simpler system and sticking to it. There isn't always a single right answer.

For marking of major capstone engineering projects which I manage we require two blind markers for each project. They both use the same criteria. These marks are then moderated with a small moderation committee of more experienced colleagues. Generally if the two marks are within a reasonable range (say one grade) it is considered OK to take the average as the final mark. If not the implication is that one or other marker may have missed something and so the moderation groups job is to determine what that is from the evidence presented by the markers and decide on the mark accordingly. Occasionally it is necessary to allocate a third reviewer who is not blind to previous marks if it isn't obvious from the previous markers evidence. There is no further process needed.