7

An assessment I'm looking at contains a few questions along the lines of:

Arrange the following in the correct sequence:

  • __: The third in sequence
  • __: The second in sequence
  • __: The fourth in sequence
  • __: The first in sequence
  • __: The fifth in sequence

What is the standard method for grading such a question?

The correct sequence of numbers to be written into the __ spaces would be 3, 2, 4, 1, 5. However, if a student misses, say, the correct starting point, but has the remainder of the sequence correct relative to each other (eg: 2, 1, 3, 5, 4), does one generally consider the entire sequence incorrect, or is partial credit of some sort more appropriate? If the latter, are there any ways to easily automate the evaluation process?

One option that comes to mind is to create a list of possible options and convert the question itself to a multiple-choice question.

Eg:

  • A. 1, 2, 3, 4, 5
  • B. 5, 1, 3, 4, 2
  • C. 3, 2, 4, 1, 5
  • D. 2, 1, 3, 5, 4

However, (a) I feel that it then becomes a bit more confusing to answer, and (b) I'm not sure if it is really assessing the same skill at that point.

  • I saw exams where you had to put four things in a specific order. Each correct thing gave you half a point. This should be reasonable. You can't prevent people from guessing, though. – Ian Feb 13 '17 at 11:50
  • 1
    @ian_itor, thanks for the comment. For clarification, if I followed your suggestion, how would you suggest that I mark the student who answers 2, 1, 3, 5, 4? None of those orders are "correct", but at least the second through fifth options are in the correct sequence relative to each other. Having multiple questions with a lower number of choices might be an option worth considering (and then just treating the entire sequence as correct or incorrect). – A Handcart And Mohair Feb 13 '17 at 14:38
  • 2
    You can use whatever grading standard you want—different rubrics are appropriate for different types of question or different levels of difficulty—but you should choose your rubric and reveal it to your students before they have to answer the question. – JeffE Feb 13 '17 at 15:19
  • @AHandcartAndMohair A half point would be awarded for the correct absolute position, not for relative positions. It is kind of arbitrary when you think of the case that a student makes one mistake and puts the first item last and gets no points, but you have to decide on something. – Ian Feb 13 '17 at 15:41
  • 2
    In some cases you might want an ad-hoc method. If the correct answer is 21354, then maybe 12354 and 23154 would be considered equally wrong by some standardized evaluation process, but maybe in terms of the actual content, one is closer than the other. Maybe the fact that 2 comes before 1 is a really basic fundamental fact that students must know, but the fact that 3 comes after 1 is a minor technical detail that isn't really so important. You might then want to give more partial credit for 23154 than 12354. – Nate Eldredge Feb 13 '17 at 18:58
3

You could give points for the longest subsequence in the correct order.
Alternatively, you could make it an open question: "Describe the process step by step, make sure to include in your answer 1, 2, 3, 4 and 5." Then grade by a few essential insights that need to be in the answer. This eliminates any easy guessing and shows you why the student misorders the process.
Also a possibility is to only let the students place 1 element of the process in the correct spot. For example:

Where in the process does 1 happen?
A. before 2
B. after 2, but before 3
C. after 3, but before 4
D. after 4, but before 5
E. after 5

Another possibility is to focus on the parts of the process that directly follow each other. Suppose the student answered "1,2,3,4,5", you could look at the pairs (1,2), (2,3), (3,4) and (4,5). Then give a point for (1,2) if 1 directly precedes 2 in the correct sequence, give a point for (2,3) if 2 directly precedes 3 in the correct sequence etc.

  • Both nice choices for me to look into further. Thanks! The open question is ideal from the demonstration of learning perspective, but I think the test itself needs to be something that can be administered without too much intervention or manual correction. – A Handcart And Mohair Feb 13 '17 at 15:14
  • Marking this as accepted. Will still probably explore more options in terms of revisiting the question structure itself. – A Handcart And Mohair Feb 17 '17 at 17:49
2

I don't know any standard method for grading such questions, but I have the impression that we get the best correlation between student knowledge and grades if we base the grading on the number of wrongly ordered pairs (or in other words, the Kendall tau ranking distance between the given answer and the correct one). A wrongly ordered pair is a pair of items (x,y), where x occurs before y in the answer, but should occur after y. So, for example,

  • (1, 2, 3, 4, 5) has 0 wrongly ordered pairs,
  • (1, 3, 2, 4, 5) has 1 wrongly ordered pair, since 3 should occur after 2,
  • (3, 1, 2, 4, 5) has 2 wrongly ordered pairs, since 3 should occur after 1 and after 2,
  • (2, 1, 5, 4, 3) has 4 wrongly ordered pairs, since 2 should occur after 1, 5 should occur after 4 and after 3, and 4 should occur after 3,
  • (5, 4, 3, 2, 1) has 10 wrongly ordered pairs.

The minimal number is 0, the maximal number is n(n–1)/2. For grading, define some function that maps the interval [0,n(n–1)/2] to points, not necessarily in a linear way.

  • This seems like an interesting and potentially workable approach. I think I might have to make up a couple of dummy responses and see what it might be like to implement it. Thanks for the suggestion! – A Handcart And Mohair Feb 13 '17 at 19:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.