13

This is probably a very broad question. It might be difficult to find one answer across all disciplines. If anyone could share their personal experiences on how they have designed tests and the thought process behind it, I would really appreciate it.

4
  • 1
    Does the theme of testing you're asking about include formative assessment (i.e. in-process tests to help students identify their own strengths) or only summative assessment (tests to rank the students against each other or a fixed standard)?
    – origimbo
    Apr 14 at 16:33
  • Hi @origimbo, formative assessments are what I am trying to know about.
    – Noob
    Apr 14 at 18:12
  • 1
    I had an epiphany while I was in college, which was that topics for a particular subject should be categorized one of three ways: either the topic is something you need to Know, it's something you need to Understand, or it's something you need to Be Familiar With. Who knows how those determinations are made, but the point is that each topic needs to be treated according to which category it fits into. As an extension, each topic's treatment in an exam setting needs to be tested accordingly.
    – John Doe
    Apr 15 at 17:12
  • I think you mostly decide by looking at your time budget for the corrections. Apr 16 at 10:14
27

In the absence of other factors, in my opinion, at least in mathematics, a free response is always the more valuable type of question. The educational value of getting shown, where precisely your argument went wrong is much higher than just being told if the final result was right or not. It also allows for a much finer grading, as I can both give partial points for an almost correct reasoning leading to a wrong answer as well as deduct points for incorrect reasoning leading to the correct answer.

That being said, a test question never happens in a vacuum and the problem usually is lack of time. This can occur in two forms, lack of working time to correct the test and lack of time during the test.

Personally, I believe the former is a bad excuse. Unless you have a 200+ people class without any TAs (in which case, clearly something else is wrong), it is always possible to correct a full-text exam in a reasonable amount of time.

This leaves the latter point. Any exam has a limited duration and it might not be possible to cover all topics in this time. My personal response would be to leave out some of the less important ones at random. As long as the topic selection is not announced to the students beforehand, they still have to study all of them and the results will be similar. But this is a point where a good argument can be made for turning part of the test into multiple-choice. In this case, the most important topics should still all be free-responses, but you could add a small multiple-choice section covering the rest. However as these are not the important part of the test and they are there to save time, I would keep them simple. (E.g. picking the right definition or "does A imply B?") After all, if marking the right answer is not substantially quicker than writing it out, then you are not saving time.

In any case I would always invert the original question. Don't take a concept and decide of it should be multiple choice, but instead decide how many multiple choice questions you have to make and then pick the questions that work best.

6
  • 8
    at least in mathematics, a free response is always the more valuable type of question. - I often ask several true/false questions in calculus for a quick conceptual check that I think evaluates different things than my free response exercises (e.g. here's one a surprising number of people get wrong: T/F int_0^1 f(t)dt = int_0^1 f(x)dx). So depending on what you want to evaluate, I think both questions have their place.
    – Kimball
    Apr 15 at 13:31
  • 1
    A perfect example of why true/false-style questions are terrible, @Kimball! There is no way that can be answered without knowing what you have arbitrarily defined as f(t) and f(x). Their integral might be the same; they might not. What understanding does this question check? Mind-reading? Memorization of notation used in your course notes? Tendency to overthink questions?
    – Cody Gray
    Apr 15 at 17:54
  • 4
    @CodyGray If you're objecting to not properly defining notation (i.e., that I have not presented a logical statement), rest assured f is qualified to be something like a continous function on [0,1] on the exam. Accordingly the answer is clear if you actually understand what a definite integral means.
    – Kimball
    Apr 15 at 18:52
  • 6
    @CodyGray I really do not understand your objection to that question. Unless I'm missing something silly, how could those integrals possibly not be the same (i.e. either both defined with the same value or both undefined)? That being said, this isn't really the place to resolve that, so consider this a rhetorical question unless you actually want to jump into a chat room or something and discuss it.
    – David Z
    Apr 15 at 21:50
  • 3
    @CodyGray Those are integrals of the same function, $f$, over the same interval. You have made precisely the error that this question is intended to detect.
    – Sam
    Apr 16 at 12:01
15

There is a problem with multiple choice questions in general: They are very hard to design well. The time you "save" in grading isn't free. You need to spend a lot of time and thought on question design and you really need to find a way to verify the questions so that you don't get negative correlations between the answers and the knowledge of students. The latter has a statistical test, whose name I've forgotten after all these years. Sorry. Maybe someone can supply it. Some auto graders will supply that item analysis for you, actually.

In a free response question, the grader can apply judgement based on subtleties. In a MCQ, that all has to be designed into the question and the provided answers. And it isn't always easy. In a free response question, a small mistake in the question itself can sometimes be worked around if the student says something "sensible". There is no such chance in a MCQ.

Ideally, there is one correct answer, usually one clearly incorrect answer. But what of the others? Ideally, again, they should be designed to capture common student misconceptions so that feedback can be given to the student on why they chose the wrong answer and what they need to do to correct their thinking. Just an "incorrect" doesn't give any insight, especially if the incorrect answer are just chosen without deep thought.

If you aren't willing to spend an hour or more designing each question and if you aren't able to validate the test, then MCQs are very dangerous. The national level standardized tests do that sort of prior analysis and post validation. They also likely have a few surprises with new questions.


The statistical test I'm thinking of measures an individuals answer for each question compared to their overall score over all the questions and aggregates it over all the test takers. If the people answering question 1, say, do worse overall on that question than their overall score on the exam, then the question itself or its answers are likely misleading. Getting this feedback as the test creator can be quite humbling. Something is wrong, either in the teaching or the testing.

If you need to give MCQs for reasons of scale or otherwise, then expect problems. Try to build in some flexibility in the usage so that it doesn't punish people who actually understand things. This is very hard if good students "waste" time in considering weird questions when they should be doing something more productive. Students aren't always very good about picking the low hanging fruit first and then going back for the harder ones. Online testing might make that more difficult, actually as going back over skipped answers might not be well supported.


I assume it is still true, but at one time education degrees in the US required a course in test design. Given that, there must be text books devoted to it.

5
  • 1
    @Ian, not the name I remember. I think it was named for a person, but it was almost 50 years ago that I last saw it.
    – Buffy
    Apr 14 at 15:55
  • And, reading wikipedia, I don't think it is quite the same, missing the aggregation step. That would seem more to be a measure for comparing the result of one question to the overall score of an individual.
    – Buffy
    Apr 14 at 16:01
  • Fair enough. Good luck figuring it out.
    – Ian
    Apr 14 at 16:02
  • 1
    @Buffy are you thinking of Cronbach's alpha? Apr 14 at 16:25
  • 1
    Hi @Buffy, Thank you very much for the write up. I just looked at a study conducted on test analysis to identify quality MCQs. I had no idea about such statistical analysis, thank you once again :)
    – Noob
    Apr 14 at 21:34
3

A great use of multiple choice questions is in formative assessments: you can test and quickly see which of your students have common misunderstandings. These questions don't work so well to assign grades, as then the student isn't interested in what they misunderstood, but are great for use in quizzes during teaching or between session to allow explanation of the errors.

As an example from a course I taught on the programming language R:

After running the following code, what will R return (answer without running)?

b<-2
a<-10
b<-20
function2<-function(){
    a<-1
    c(a,b)
}
c(function2(),a,b)
  • 10,20,10,20

  • 1,2,10,20

  • R will produce an error - Error: object ‘b’ not found

  • 1,20,10,20

Each of the answers has its own reason for being chosen: 1 shows a misunderstanding of using local scope, 2 shows a misunderstanding of reassignment, 3 shows a misunderstanding of scoping going to global, and 4 is correct.

This kind of multiple choice question is obviously more work, but is much superior to a free response. I would do these with a show of hands and ask students to argue their point - much more difficult to do without the red herring answers.

2
  • But isn't the main reason for these so you can ask a non-graded question and students can anonymously use their A/B/C/D clicker and you get a feeling if they're following along? This is a not-so-bad MCQ, but on a graded test how is it better than free? Apr 16 at 3:36
  • What's the answer? Never used R and can't figure out how the recursion would work!
    – mkst
    Apr 16 at 9:38
2

True or False?

Since there's a lot of intersubjectivity here, and I agree that multiple choice is not optimal for assessment, I notice that one aspect of multiple choice has been left out, so I'll play Devil's advocate: true/false questions.

While it does depend on your content area, adding a few well-written T/F questions to make up a part of an evaluation can be useful. It can save students time of writing out full answers better devoted to FRQs while still showing that they retained certain facts. It is up to the author of the test (that's you) to decide what merits a free response in a given subject matter. For example, in teaching world language courses, it can be a simple way to evaluate if a student has retained key cultural, geographical, or historical topics. Keep the statements simple and to the point (even drawing attention to negative statements). Even then, it can be better to ask a short free response question to eliminate guessing or cheating. Take things like really big dates or facts that students either know or don't (i.e. What year was the Battle of Hastings?), which do not benefit from a multiple choice question. Asking it as T/F doesn't ward off guessing and statistically encourages it. One deterrent for this is to use a free response or short answer question to spot check a random T/F question; this also encourages differentiated learning and critical thinking. Unless you are trying to get a computer to grade the entirety of an exam (don't; people take the tests and people can grade them), there's no more work for you to read "1066" than to see true or false circled. It is a better example of the student's work and retention. I would go with the short answer question there.

Think way back to your earliest spelling tests and which was the better assessment: did you show what you were thinking and how you were learning when you wrote out the word your teacher said or when you saw four words with three spelled close but wrong?

Overall, free response is the best insight and evaluation of your students; it leaves more room for the individual and less for testing errors. Of course, some disciplines and depths can fit in multiple choice better than others, but reading FRQs is not as tedious as writing multiple choice and a gives a better demonstration of nuances. I suggest sprinkling in just a handful of T/F for differentiation and time management.

3
  • 2
    Memorization-style questions about "big dates or facts that students either know or don't" don't generally deserve being asked on exams at all. Who cares that the Battle of Hastings was in the year 1066? What matters is the significance of the Battle of Hastings. Maybe you can think of a good way to assess understanding of that in a T/F question. I can't, and never have been able to, which is why I never wrote exams with T/F questions. (Having the student write out "1066" doesn't help, either. It just optimizes for the worst kind of rote learning.)
    – Cody Gray
    Apr 15 at 18:02
  • I've always considered T/F Q's as a version of MCQ. "King Edward died during the Battle of Hastings[T/F]". or "The Battle of Hastings was caused by: A) death of Kind Edward, B) ...". I've starting writing a few MCQ's and realized T/F is about the same but quicker and takes less space. You don't want to mix T/F and MCQ -- it confuses and distracts students. Apr 15 at 18:32
  • The UK Maths GCSE papers do this quite well - they are largely composed of free-answer questions with a handful of multiple-choice questions (mostly to get the students relaxed at the start), but also tend to include a few true/false questions that have the form "Is this right? [true/false boxes]. State your reason [free response section]. You get a lot of the quick marking benefits, which gives a nice middle-ground.
    – Sam
    Apr 16 at 15:22
1

I like multiple choice for Q's where you'd otherwise have to explain the sort of answer you want too much or you think they might misunderstand:

"What operation has the highest precedence? +, ^, *, /". I'm looking for a binary operator, not () or unary minus. Limiting the answers is the easiest way to specify that. "What is the formal name for a folder? Directory, file-bucket, Organizer, ...". Students can spin out on a term like "formal" and "the formal term that would be in the manual" is long and not all that much better. "Why did Virginia object to the 1st constitution? A, B ... ". There were lots of reasons. Writing "the best reason" tends not to help much. Choosing from a list with one clearly best answer is safer.

Of course, it depends on what you want to ask and how much detail. We could just as easily have them show the order of operators in an equation; or ask "briefly list 3 reasons Virginia...".

8
  • Thanks a lot @OwenReynolds. It does make sense. I have seen particularly among high school teachers a distrust towards MCQs, saying "some students always cheat". If you don't mind me asking, is your answer presented in terms of evaluating a college level history class or math class?
    – Noob
    Apr 15 at 1:21
  • Hmmm...like they cheat by flashing hand-signs for A, B ... D? No, not history or math, but yes college. I remembered the trick, but not any specific Q's. In fact, my greatest use of MCQ's was half of a Friday afternoon final for 400 students w/grades due Monday. Apr 15 at 1:57
  • I'm hella confused by your formatting Apr 15 at 2:26
  • @OwenReynolds, Yes hand-signs. Also it's very easy to cheat for MCQs
    – Noob
    Apr 15 at 7:30
  • "Choosing from a list with one clearly best answer is safer." For the lazy or ill-informed educator. The rest don't need to constrain the students' choices arbitrarily. They can just assess the quality of the student's actual answer: is what they wrote a valid reason why Virginia objected to the Constitution? It doesn't need to match up with the wording given in a textbook or course packet.
    – Cody Gray
    Apr 15 at 18:05
1

There's nothing wrong with a healthy mix!

Though your question is very broad and the answer changes depending on the age group of students and their abilities, there is nothing wrong with a good mix (suited specifically to both of the possibilities above,) though in my answer I've included some facts and provided the most conclusive answer that I could.

While an open-ended question allows for the teacher to see exactly where the student went wrong and how to correct it, and also prevents a bad habit of guessing that often happens on multiple-choice, multiple-choice often has it's advantages as well.

For one, it makes for a much less tedious and painful, frustrating test. Sometimes students with special learning needs or learning disabilities may not be open to such a long test in this way, and if you are working with younger elementary-aged students, they do not have the attention span and it is often noticed that the quality of work goes down the more questions that are being asked.

However, know the abilities of your students. For High School/College level, more open-ended responses may be the way to get the test that gives you valuable information as a teacher. If you are teaching elementary students with a low attention span, maybe use one or two open ended questions with the rest being multiple choice.

Again, it really depends on the overall ability of the students. The starting ratio (for upper elementary students) should be about 50/50 (ex. 5 open-ended, 5 multiple choices) for a test with ten questions, but this ratio can be adjusted for the individual needs of your students or depending on the age.

1
  • 1
    Thanks! One example I can think of is the AP exams where there is a good balance of MCQs and FRQs. My secondary school exams were also a mix of the two.
    – Noob
    Apr 17 at 7:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.