If I use an auto-grading script and allow students to see their would-be grade based on the auto-grader, is it wrong to change their grades after the due date if I decide to change the auto-grader? I'm curious what others think because on the one hand students could go back and change their submissions as many times as they liked prior to the deadline, and many students took advantage of this to do as well as possible. Now that I've changed the script many students now have a zero (including a few who previously scored perfectly). On the other hand, the issue that these students had was a result of not following an instruction I gave out, which I overlooked while writing the original auto-grader. So is it wrong to change the grades or am I justified?
If any student feels disadvantaged by this, then you will have an uproar and complaints to administration. You describe a system in which they depended on the actual thing you built not some "instruction" you gave.
I would guess that you are stuck with the thing for this group of students, both practically and ethically. Otherwise grading will seem chaotic and unfair to them. Your reputation might suffer both with them and with your boss.
This sounds more to me like a case of releasing untested or insufficiently tested software.
One way out would be to void the results of the autogravder for anyone disadvantaged and give them one additional attempt to submit. But that could also be a problem because of time constraints on other things in your course and in their other courses.
Live and learn. But first, be fair.
- Now that I've changed the script many students now have a zero (including a few who previously scored perfectly).
- ... because on the one hand students could go back and change their submissions as many times as they liked prior to the deadline
I don't see how (2) matters here.
Here's what's effectively happening here: If I was in your class, I submitted an assignment, and you told me that it's perfect, I obviously wouldn't revise it later. You can't at the last moment decide that you didn't do the right marking and give me a zero. That's not how it works. If you'd told me beforehand that the work needed to be revised, I would have done so.
But you can't say that you're giving me a zero after telling me it's a perfect score, and after the deadline, and not allow me to revise it.
If you go through with changes to the auto-grader, your students will lose any respect they have for you. As will your colleagues, probably, since they'll think that you tried to take the easy way out by writing an auto-grader and then messed it all up.
I guess whether it's strictly speaking "wrong" depends on how you communicated before (i.e., could a reasonable student interpret your grading script as part of the assignment spec, or the ground truth about how your instructions are to be understood?). However, pretty much independently of how you communicated, you will get a lot of backlash if you provided a grading script that students could use which gave them full points, and for the actual grading you change the script so that the same solution gets 0 points. Students will not be pleased, they will complain, and more likely than not they will have a case to do so.
In essence, think of it that way - if you manually "pre-graded" solutions and told groups that everything is ok, would you then consider it fair to change your mind and give the same groups 0 points? If not, is it so different if the grading is done through an automated script?
Let the grades stand for this year, and improve your grading toolkit for the next iteration.
My suggestion is to:
- Give them a grade that matches the initial auto-grader.
- Also tell them that there's something in the requirements that the auto-grader failed to check for.
- As part of the next assignment, give them a task that builds upon the current one and can only be accomplished if they also meet that specific requirement you initially didn't check for.
- Make sure that your updated auto-grader checks for every requirement you care about in their submissions, including the one you previously forgot to include and the ones that pertain to the new task.
Students who didn't follow that specific instruction will now have to update their solutions accordingly, before implementing the new task.
This approach doesn't punish anyone for failing to meet a requirement that your auto-grader didn't enforce and offers everyone the same opportunity to learn and to prove that they master the topic of the assignment.
You seem to have put yourself somewhat between a rock and a hard place.
The rock is when you keep the auto grader at its previous value. This approach disrespects your intent to have performance based on students doing exactly what you ask. The hard place is when you change the auto grader to the newer value. This approach portrays that your grading metrics are subject to last minute changes even when those changes are likely fully justified.
YOU made a mistake. Admit it and don't punish students for it. Do not change your auto-grader AFTER the grades have already been posted. This approach will leave a sour taste for everyone all around. Consider, do you really want to give the impression that students have to be in competition with each other for a better grade when you make a mistake?
When you really feel that you need to rebalance the grades in some way, offer an opportunity for extra credit on the assignment. But make that opportunity open only to students who did exactly what you asked in the first place.
This is more of an extended comment on the effect of auto graders and especially problematic ones. First an example that is a bit extreme, just to make the main point.
Suppose I give a multiple choice test that is auto graded but also permits the student to take the exam multiple times and only submit one of them.
What does that emphasize? In fact, I doubt that it promotes learning at all, and emphasizes that the teacher wants answers even if at the expense of learning. The student is encouraged to just guess and re guess until they "get it right".
The problem is not just the obvious one, but the fact that such a system that seems to require work from the student provides no actually useful feedback to them on whether they have learned the proper lesson. Student work should always come with effective feedback, especially when the student has made errors. The feedback should attempt to give the student an idea about where their thinking when astray so that they can do better in the future.
I can't claim that the situation of the OP has these characteristics, so haven't made this a proper "answer" here. But it is something to think about for anyone wanting to incorporate such a thing into an educational system. Think beyond the obvious about what unintended consequences there might be. Do this prior to implementation, of course.
Education requires reinforcement and feedback. Make sure that your system provides both. And the feedback needs to be effective, not just "yes" or "no".
If a specific test case was marked as passing or failing in the sample validation, it should still be marked the same in the final way that you grade. Ideally you would also have text explaining that the sample validation only tests SOME test cases and the final grade will be based on a much larger set of test cases or criteria, but I suspect it's a moot point in this specific case because it sounds like you changed the result on some of the existing tests. (However, you should include that discussion for future assignments.)
I'm using the term sample validation because your "auto-grader" that you give to your students should never be the full grading criteria. If nothing else, a student could turn any assignment into a map that takes the input from the auto-grader and returns the correct output by "magic", which would be acceptable if that was literally all of your grading criteria.