We teach a large graduate class with many teaching assistants (TAs), who are involved in grading 4-5 assessments (each contains a large amount of code, and a 4-page report corresponding to the solution).
Each time we grade, it takes an extremely long time-- we first carefully create a rubric, based upon which the TAs evaluate each others' solutions; next we review these grades and come to a consensus on individual line items in the rubric. We also blindly grade a single student submission at a time and re-calibrate the rubric and it's application till we have consensus.
Despite doing all of this, we end up having non-trivial variations in the mean, median, min and max grades assigned by each TA. We have hundreds of students and dozens of TAs, and it isn't feasible to have multiple grades per student submission.
Last Spring, this led to a significant number of grading iterations for each assignment, and we eventually ended up averaging across iterations, rather then converging to a common grade for each student.
We're trying to automate the monitoring and evaluation of grading tasks on our grading portal so that outliers/problem cases can be reassigned, and the lead instructor can identify if a TA is not grading well. Is there any research or anecdotal experience with such grading issues in classes that you have taught/assisted in (and any useful countermeasures)? If not, is there a systematic way where grades can be curved based on the grader, in a way that is fair?