What are some strategies for using automatic grading technologies to get students to express their reasoning?

Question

We currently use an online quiz tool to give students questions in some of our engineering classes. These questions might be simple or very challenging, but at the end of all of them the students are required to put in "the answer" - a numeric value, or rarely a formula they've derived. These quizzes are run every week, and are basically a sneaky way of getting students to do homework problems. In that, they are very successful. The quizzes are basically open-everything - they can google things, talk amongst themselves, ask us anything etc.

However, we are becoming aware that this type of quiz leads students to race to find the right equation, plug in the right numbers, and move on. As an example, one of the questions from a recent circuits quiz was relatively straightforward in terms of the circuits concepts, but required careful planning of the math in order to avoid any errors. We've been tracking the questions that we are asked during the quiz, and almost universally the students 100% understood the circuits concepts but were making math errors. This is actually a good thing - we want to encourage students to learn to structure their reasoning in a careful, well-planned way, so we are OK with students struggling at it. However, the question doesn't allow for any way to give feedback on it. The answer either is or is not numerically correct.

Obviously we can have the students submit to us and we can evaluate them directly, but that isn't the point. An automatic grading system isn't ever going to be able to give students the same level of feedback as a human TA, but I feel that we are not using it to its fullest potential to give students feedback on their process.

What are some strategies we can use for question design in automatically graded quizzes that might help here?

Answer 1

I think you've hit the limit on what automatic grading technology can achieve at this current point in time. Generally, this technology is only going to allow you to automatically evaluate questions with pre-defined answers. This doesn't work for theory or structuring. It probably isn't going to, either.

However, you can use it to evaluate concepts. I'm not exactly sure how your class works, but here are some example concept questions.

Given [Situation X], what methodology would you use to solve [Problem Y]?

The answer would be from multiple choices.

You also combine this with over method comprehension. Let's do true/false.

[Method X] is the process of..... (True/False).

From there, you can then start going through some actual calculation questions. If structured correctly, you can deduce the information you're trying to achieve.

• If the student does well on the conceptual questions and bad on the math questions, they likely made some arithmetic errors.
• If the reverse is true, the student probably just memorized the formulas.

These situations aren't always going to be true, though. They will never be as accurate as a human evaluating a student's process. But the estimates may still prove useful.