When should the grading scheme for quizzes and exams be finalized?

Question

I am currently working as a graduate teaching assistant for a semester-long introductory physics course, aimed primarily at Biology/Pre-Med undergrads. The students usually have a short quiz every two weeks in a discussion section, along with two prelims (mid-terms) and one final exam. The TAs make the grading schemes for their own quizzes as each discussion section has a different quiz.

One thing that I find very strange with the grading (quiz or prelim) is that the grading scheme is made after seeing the answer sheets submitted by the students, not before. The intention behind doing so (at least for the quizzes) is to assign less points to the harder questions and more to the easier ones so that the average is roughly 7.5/10 (this is a soft rule). Similarly, on the prelim, the points for individual sub-questions were decided after conducting the exam and seeing a few answer sheets (points for each question were fixed beforehand).

1. Is this a standard practice for introductory physics courses in the US?

2. What are the benefits of this practice over deciding a grading scheme beforehand?

From an answer describing a good grading system:

At minimum, a good grading system should meet three criteria:

1. it should accurately reflect differences in student performance
2. it should be clear to students so they can chart their own progress
3. it should be fair

On the first two counts, making the grading scheme afterwards seems to be a bad choice:

1. Accurately reflecting differences in student performance → making the grading scheme later may bias the grader towards assigning higher points for conceptually easy questions instead of more challenging ones.
2. Clear to students → the points for individual sub-questions are not assigned beforehand so students cannot judge how much time they should devote to different problems.

Answer 1

This seems perfectly reasonable to me. In this type of freshman physics course, there are some very common difficulties that students have:

• Physics is counterintuitive, e.g., students often intuitively believe in some kind of half-baked aristotelianism, and this belief can be resistant to instruction.

• Students try to solve problems by shopping for a formula that has the right letters in it, and then plugging numbers into the formula. They don't think about what the formula means, when it's true and when it's false, etc.

This is all very well documented in the pedagogical literature. The best summary I know of, although it's out of date, is in Mazur's book Peer Instruction. Mazur shows a lot of evidence that physics instructors are in denial about what their students are really learning and about the (in)effectiveness of traditional teaching methods. There is a huge gap between how physicists think about their subject and how their students think about it.

A lot of these issues are qualitatively different than in other fields, such as mathematics. E.g., I don't think students have deeply held intuitive preconceptions about derivatives and integrals before they walk into a freshman calc course. In a course like freshman calc, a lot of what the students are learning to do really is just algorithmic -- there's a reason it's called calculus.

For these reasons, it can be extremely difficult for an instructor in a freshman physics course to write appropriate exam questions. Results can be really horrible if you write questions that require conceptual understanding, multistep problem-solving, symbolic rather than numerical calculation, or interpretation of symbolic results. You can write problems that just require plug-and-chug, but then you're dumbing down your educational standards.

Because of the gap between how instructors think about physics and how most students think about physics, there is a real danger when setting an exam that you will be taken by surprise by how horrible the results are. This is particularly likely to happen with inexperienced instructors. Fiddling with standards after taking a look at the answers is IMO not an unreasonable way to prevent this kind of meltdown. Especially for a non-tenured instructor who can't afford bad teaching evaluations, I think it's pretty understandable to do this. As time goes on, people usually get a better feel for what kinds of questions will be hard. If I were evaluating someone for tenure, I would rather see them doing the kinds of things described in the question than see them writing plug-and-chug exams.

There is one thing that does seem a little odd to me about the specific situation described in the question. If I'm understanding the question correctly, the instructor in charge of the course seems to have delegated the writing of some of the lower-stakes quizzes to TAs, but the instructor will still be the one writing the final. However, there doesn't seem to be any mechanism for making sure that the standards and style are consistent throughout the course. Ideally students would be doing homework that is similar in style and difficulty to the quizzes, prelims, and final. The instructor should be setting the tone as to what should be the mix of easy and hard, concepts and mathematics, numerical and symbolic. If that isn't happening, then something is wrong.

Daniel R. Collins wrote in a comment:

In future semesters adjust the tests so that's not needed. Personally, I don't find it hard to dial in an appropriate difficulty level after the first semester.

If these questions are being written entirely by TAs, with almost no supervision by the instructor, then it seems unlikely that there is any institutional memory that would allow this type of adjustment to occur over time. If that is the case, then the serious problem is lack of supervision, guidance, and institutional memory -- not a particular practice in grading quizzes.

Answer 2

This seems like a variation of norm referenced grading aka grading to the curve. What is unique in your question is that the TAs are not using the student's responses to set the curve for the overall score of the quiz but rather to set a curve for each individual question. This is strange in my personal experience but I cannot say definitively that it is an unacceptable practice. However, probably no one with thorough training in assessment in education would do this.

One of the concerns you mentioned was fairness in the grading. One way to address this is to clearly communicate how quizzes and exams are assessed and graded. If you explain the current system it will help significantly. If students know what to expect they may disagree but it reduces the risk of complaints of unfairness

Answer 3

There are definite advantages to this scheme, but I see one big potential pitfall.

The advantages: sometimes a question you pose has a "garden path" wrong answer where students can follow what seems like a perfectly good line of reasoning, and end up not getting it correct. Maybe attacking it from this approach would work in principle, but you end up with horrible equations you need to solve. Whether a student gets these questions right might be rather random, depending on which approach they happen to think of first. If you reduce the number of points these problems count for, the grading will be fairer.

The potential pitfall: it will be very tempting to look at what the two or three best students in the section did, and use this to calibrate the questions. This gives an enormous advantage to the students you think are the two or three best students in each section, and you should try to avoid doing this.