17

If I use an auto-grading script and allow students to see their would-be grade based on the auto-grader, is it wrong to change their grades after the due date if I decide to change the auto-grader? I'm curious what others think because on the one hand students could go back and change their submissions as many times as they liked prior to the deadline, and many students took advantage of this to do as well as possible. Now that I've changed the script many students now have a zero (including a few who previously scored perfectly). On the other hand, the issue that these students had was a result of not following an instruction I gave out, which I overlooked while writing the original auto-grader. So is it wrong to change the grades or am I justified?

  • 68
    How did a change in the script take some people from 100% to 0%? – Azor Ahai Oct 10 at 20:52
  • 12
    Fwiw in my classes the autograder only ran about 20 test cases, and provided immediate feedback. But once the assignment closed, it ran 100+ cases. So students get feedback during the assignment, but are also inventivized to check their work even after hitting 100% in the live autograder. I still remember one of my missed full test cases for Virtual Monopoly - my submission did not correctly handle a chance card that moved you backwards past Go. – MooseBoys Oct 11 at 7:45
  • 18
    On the other hand, the issue that these students had was a result of not following an instruction I gave out -- Is this instruction capricious in nature, like you must have a comment saying orgalorg or something sincerely so fundamental to the assignment that they demonstrated no mastery of the topic? – trognanders Oct 11 at 10:39
  • 34
    The first rule of grading is "Students should not suffer from instructors' mistakes." – JeffE Oct 11 at 14:52
  • 19
    @forty_tw0 So, in other words, the jump from 100% to 0% is a result of your tool's output, rather than any actual lack of topic mastery - while they didn't use exactly the tool you supplied in the spec, they did use the tool you supplied as part of the submission (and they are using a valid C compiler - declarations being required at the top of a given block scope were last in the C89 standard) – Delioth Oct 11 at 20:44
64

If any student feels disadvantaged by this, then you will have an uproar and complaints to administration. You describe a system in which they depended on the actual thing you built not some "instruction" you gave.

I would guess that you are stuck with the thing for this group of students, both practically and ethically. Otherwise grading will seem chaotic and unfair to them. Your reputation might suffer both with them and with your boss.

This sounds more to me like a case of releasing untested or insufficiently tested software.

One way out would be to void the results of the autogravder for anyone disadvantaged and give them one additional attempt to submit. But that could also be a problem because of time constraints on other things in your course and in their other courses.

Live and learn. But first, be fair.

  • 5
    Totally agreed: ++1. Never let a software dictate how you give out marks, unless the students had the opportunity to gauge themselves against it (which they have). Changing this after the fact is pulling the rug under them. – Captain Emacs Oct 10 at 21:33
  • 9
    I would add as an addendum that if you are transparent about the fact that you made a mistake and you give students the chance to amend their submissions against your amended autograder, that would probably be okay. I majored in comp sci, and we had autograders for some small coding assignments. mistakes happen. given the ambiguity of instructor's word versus auto-graded score, you really cannot fault them for going by what the autograder told them and if you decide to retroactively change the standard, it's only fair to let them retroactively redo the assignment. – Wug Oct 11 at 8:18
  • 9
    And do tell the students about it. "There was a bug in the autograder. While your grades stand, it didn't account for XYZ. Therefore XYZ will be in an upcoming test instead." – Gloweye Oct 11 at 14:38
  • 4
45
  1. Now that I've changed the script many students now have a zero (including a few who previously scored perfectly).
  2. ... because on the one hand students could go back and change their submissions as many times as they liked prior to the deadline

I don't see how (2) matters here.

Here's what's effectively happening here: If I was in your class, I submitted an assignment, and you told me that it's perfect, I obviously wouldn't revise it later. You can't at the last moment decide that you didn't do the right marking and give me a zero. That's not how it works. If you'd told me beforehand that the work needed to be revised, I would have done so.

But you can't say that you're giving me a zero after telling me it's a perfect score, and after the deadline, and not allow me to revise it.

If you go through with changes to the auto-grader, your students will lose any respect they have for you. As will your colleagues, probably, since they'll think that you tried to take the easy way out by writing an auto-grader and then messed it all up.

  • The third paragraph should be bold and the only part of an answer needed. Sometimes I really wonder where common sense is. – Hakaishin Oct 11 at 15:41
  • @rahuldottech: You bolded the 4th paragraph. – Brian Oct 11 at 17:34
  • @Brian RIP me, I should get more sleep. I've bolded both now, I think that should be okay. – rahuldottech Oct 11 at 17:35
  • > But you can't say that you're giving me a zero after ... For the fairness to all of the students in the class, instructors have a right if not a standing obligation to fix any mistakes that they may make in grading an assignment as soon as those mistakes are discovered. An instructor is only obligated to allow revisions to regarded assignments when such a policy was stated in the syllabus or on the assignment itself. So, the OP may very well be able to do exactly what you stated he/she can't. But, even when he/she can, the more important question is should he/she? – Jeffrey J Weimer Oct 11 at 22:51
  • 3
    @JeffreyJWeimer ah, yeah. When I said "can't", I guess what I meant was that "it isn't morally right to" – rahuldottech Oct 12 at 6:00
10

I guess whether it's strictly speaking "wrong" depends on how you communicated before (i.e., could a reasonable student interpret your grading script as part of the assignment spec, or the ground truth about how your instructions are to be understood?). However, pretty much independently of how you communicated, you will get a lot of backlash if you provided a grading script that students could use which gave them full points, and for the actual grading you change the script so that the same solution gets 0 points. Students will not be pleased, they will complain, and more likely than not they will have a case to do so.

In essence, think of it that way - if you manually "pre-graded" solutions and told groups that everything is ok, would you then consider it fair to change your mind and give the same groups 0 points? If not, is it so different if the grading is done through an automated script?

Let the grades stand for this year, and improve your grading toolkit for the next iteration.

6

My suggestion is to:

  • Give them a grade that matches the initial auto-grader.
  • Also tell them that there's something in the requirements that the auto-grader failed to check for.
  • As part of the next assignment, give them a task that builds upon the current one and can only be accomplished if they also meet that specific requirement you initially didn't check for.
  • Make sure that your updated auto-grader checks for every requirement you care about in their submissions, including the one you previously forgot to include and the ones that pertain to the new task.

Students who didn't follow that specific instruction will now have to update their solutions accordingly, before implementing the new task.

This approach doesn't punish anyone for failing to meet a requirement that your auto-grader didn't enforce and offers everyone the same opportunity to learn and to prove that they master the topic of the assignment.

3

You seem to have put yourself somewhat between a rock and a hard place.

The rock is when you keep the auto grader at its previous value. This approach disrespects your intent to have performance based on students doing exactly what you ask. The hard place is when you change the auto grader to the newer value. This approach portrays that your grading metrics are subject to last minute changes even when those changes are likely fully justified.

YOU made a mistake. Admit it and don't punish students for it. Do not change your auto-grader AFTER the grades have already been posted. This approach will leave a sour taste for everyone all around. Consider, do you really want to give the impression that students have to be in competition with each other for a better grade when you make a mistake?

When you really feel that you need to rebalance the grades in some way, offer an opportunity for extra credit on the assignment. But make that opportunity open only to students who did exactly what you asked in the first place.

2

I changed the compiler used (students submit the .c file and the auto grader compiles it) and with the new compiler a parse error occurs if any declarations are made after any non-declaration statement(s), so some students' code no longer compiles.

The student errors are rooted in a language version technicality. It is not ethical to give anyone a zero, or even a lower grade really unless it is from more comprehensive test cases. A minimum should be that every test case that passed in the student grader should pass in the official grading version.

It is fine to specify an exact target for programming assignments, but the grader must be consistent with the specification. Many CS programs provide ssh access to a shared student work linux machine or a standard virtual machine image that represents the exact grading environment.

Being consistent with the student grader allows you to be much more strict with the grades provided by the auto grader, otherwise you are open to regrade requests based on trivial differences between compilers or language versions.

0

This is more of an extended comment on the effect of auto graders and especially problematic ones. First an example that is a bit extreme, just to make the main point.

Suppose I give a multiple choice test that is auto graded but also permits the student to take the exam multiple times and only submit one of them.

What does that emphasize? In fact, I doubt that it promotes learning at all, and emphasizes that the teacher wants answers even if at the expense of learning. The student is encouraged to just guess and re guess until they "get it right".

The problem is not just the obvious one, but the fact that such a system that seems to require work from the student provides no actually useful feedback to them on whether they have learned the proper lesson. Student work should always come with effective feedback, especially when the student has made errors. The feedback should attempt to give the student an idea about where their thinking when astray so that they can do better in the future.

I can't claim that the situation of the OP has these characteristics, so haven't made this a proper "answer" here. But it is something to think about for anyone wanting to incorporate such a thing into an educational system. Think beyond the obvious about what unintended consequences there might be. Do this prior to implementation, of course.

Education requires reinforcement and feedback. Make sure that your system provides both. And the feedback needs to be effective, not just "yes" or "no".

  • Agreed on retaking multiple-choice tests. But re: "Student work should always come with effective feedback", a counter-argument is that many instructors are users/proponents of the "check homework for completion only" policy, e.g.: matheducators.stackexchange.com/questions/13423/… – Daniel R. Collins Oct 11 at 14:41
  • For a programming class an automated tester with a good report and rapid feedback cannot be a wonderful instructional tool. – trognanders Oct 14 at 4:53
0

If a specific test case was marked as passing or failing in the sample validation, it should still be marked the same in the final way that you grade. Ideally you would also have text explaining that the sample validation only tests SOME test cases and the final grade will be based on a much larger set of test cases or criteria, but I suspect it's a moot point in this specific case because it sounds like you changed the result on some of the existing tests. (However, you should include that discussion for future assignments.)

I'm using the term sample validation because your "auto-grader" that you give to your students should never be the full grading criteria. If nothing else, a student could turn any assignment into a map that takes the input from the auto-grader and returns the correct output by "magic", which would be acceptable if that was literally all of your grading criteria.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.