9

We teach a large graduate class with many teaching assistants (TAs), who are involved in grading 4-5 assessments (each contains a large amount of code, and a 4-page report corresponding to the solution).

Each time we grade, it takes an extremely long time-- we first carefully create a rubric, based upon which the TAs evaluate each others' solutions; next we review these grades and come to a consensus on individual line items in the rubric. We also blindly grade a single student submission at a time and re-calibrate the rubric and it's application till we have consensus.

Despite doing all of this, we end up having non-trivial variations in the mean, median, min and max grades assigned by each TA. We have hundreds of students and dozens of TAs, and it isn't feasible to have multiple grades per student submission.

Last Spring, this led to a significant number of grading iterations for each assignment, and we eventually ended up averaging across iterations, rather then converging to a common grade for each student.

We're trying to automate the monitoring and evaluation of grading tasks on our grading portal so that outliers/problem cases can be reassigned, and the lead instructor can identify if a TA is not grading well. Is there any research or anecdotal experience with such grading issues in classes that you have taught/assisted in (and any useful countermeasures)? If not, is there a systematic way where grades can be curved based on the grader, in a way that is fair?

  • 4
    Just to be clear - the assignments are randomly assigned to the graders, and the TAs aren't themselves in any way involved in the teaching of the material/students they are grading, right? As it is this seems to be already be the most sophisticate grading system I've personally encountered outside standardized testing systems (well, and psychology research). Inter-rater reliability is a popular issue in psychology generally, but I'm not immediately aware of research that applies to this situation - and it usually assumes multiple graders per assignment. – BrianH Jun 30 '16 at 1:35
  • @BrianDHall, assignments are random and anonymized. Some of the TAs are more involved in the teaching and assignment building process than others. All of them have to solve the assignments before we hand it out to students. Even specific keywords (like inter-rater reliability) to search for are appreciated. – Jedi Jun 30 '16 at 1:42
4

I am not entirely sure if this is an "answer" or a "comment" but I'll supply it as an answer.

First off, I'm answering from the perspective of academia -- not from the perspective of perfect statistical analysis or experiment design or anything like that. So put away any screams that this fails some sort of T test or does not produce a normal distribution -- those aren't the goals of academic work in general.

I think you might be trying to solve this problem the wrong way in all honesty. You state:

We also blindly grade a single student submission at a time and re-calibrate the rubric and it's application till we have consensus.

and then

Last Spring, this led to a significant number of grading iterations for each assignment, and we eventually ended up averaging across iterations, rather then converging to a common grade for each student.

This sounds like an immense amount of work to give students grades. But worse than that, it's clear the earlier work doesn't do anything for you since you had to take the consensus-achieved "re-calibrated rubric" and then if I'm reading correctly redo and jig together a bunch of things to produce grades.

If it seems like an immense struggle to come up with grades, then at least on of the following seems true to me:

  1. You and your team are suffering from a strong case of OCD.
  2. You're probably not achieving real consensus on how to grade things so much as minimal acquiescence on a single answer.
  3. You and your team are fighting against gravity -- i.e. you are trying to counteract a feature of human grading that you should just accept and work around.
  4. A misundertanding of the nature of academic work and grading (for more on that topic see this question and especially this answer: https://academia.stackexchange.com/a/31526/20058).

I see three potential solutions:

Rather than try to QC to perfect grade normalization, accept that individual grading differences are in-eradicable and make it so students' work is graded by a distribution of graders and that grades are not so subject to this fluctuation as to be questionable. For instance, have each grader look at a sample of 5 "standards" (an A, B, C, D , F) and see what grades they assign to them. Use this to categorize graders as severe, neutral, and soft and make it so everyone gets a fair mix of graders.

And/or make the rubric explicitly clear to the point where individual differences don't matter. i.e.,

"one point for a program that complies, one point for a program that
executes without crashing, one point for a program that produces the correct output, two points for using a recursive function, one point for mentioning "iterative sort" / "iterative sorting" in the description.

And/or figure out where the individual grading differences happen and minimize the importance of these for actual grades.

  • Great answer, really. As I said here, uncertainty in grading is unavoidable, and both professors and students should put up with it. – Massimo Ortolano Jun 30 '16 at 7:39
  • While you're both probably right, over the 4 semesters that this course has been offered, we have evidence that the steps (grader training, rubric calibration, outlier grade moderation) have led to "fairer" grades with fewer grading tasks per submission. We're trying to now incorporate all the "best practices" that we can find into our portal; individual instructors can then pick and choose which steps they want to include in their grading workflow. – Jedi Jun 30 '16 at 10:01
  • 1
    Your comment changes my image of what you're trying to do slightly ... but not entirely. In the comment, you're basically indicating that all of this work was a project to optimize the grading process as a pedagogy research project of sorts. You claim fairer" grades with fewer grading tasks per submission. But if you factor in all the time you did on the optimization surely it's not less grading time and if you have to have all these pound out sessions to figure out how to grade things, surely not fewer grading tasks. – virmaior Jun 30 '16 at 10:15
1

I agree with the previous answer in that you do seem to be making more work out of this than is necessary.

For the programming part of the assignment I would suggest defining very specific marking criteria - if the program successfully does this then you get this mark. This should be easy to grade and could even be automated. Some aspects of programing style also could be marked with precise criteria.

For more subjective aspects - the report or perhaps some aspects of code style you probably don't want to be quite as specific. You are going to get variation between markers no matter what you do. However I would suggest that going around iterations as you are saying that you do is more likely to lower confidence in grading than having a simpler system and sticking to it. There isn't always a single right answer.

For marking of major capstone engineering projects which I manage we require two blind markers for each project. They both use the same criteria. These marks are then moderated with a small moderation committee of more experienced colleagues. Generally if the two marks are within a reasonable range (say one grade) it is considered OK to take the average as the final mark. If not the implication is that one or other marker may have missed something and so the moderation groups job is to determine what that is from the evidence presented by the markers and decide on the mark accordingly. Occasionally it is necessary to allocate a third reviewer who is not blind to previous marks if it isn't obvious from the previous markers evidence. There is no further process needed.

  • You're right, we have been splitting hairs, but it has worked for us (see previous comment) to some extent. We're now just gathering techniques that we can add to our grading portal; we already support multiple graders and normalization. What we're light on is post-grading techniques, i.e. once individual TAs are done grading, we are trying to visualize biases and (hopefully) automatically "adjust" for these before releasing grades to students. – Jedi Jun 30 '16 at 10:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.