Should we ask ambiguous questions on an exam?

Question

I'm TAing an undergraduate course at a university in the US, and had a disagreement with the professor teaching the course about a question in an exam. Here's an abstract version of the relevant part:

Select the single most appropriate method, out of the list {A, B, C, ...}, to solve problem X.

Both A and B are appropriate to solve X, and objectively superior to any other method on the list. Weak arguments can be made for either of them being more appropriate than the other. After a debate with the professor, we agreed that the question is ambiguous.

My opinion is that we should not be asking such a question. I think students might waste a lot of time trying to find their presumed mistake, since the question asks for a single most appropriate method, and there isn't one. Therefore, I think we should either remove it, remove one of the two methods, or rephrase it to

Select a single appropriate method...

The professor argues that the ambiguity is not an error on our part. They suggest we'll give full credit to either choice of A or B, and say that students shouldn't obsess over this minor detail at the expense of other parts of the exam.

I feel like this is problematic in terms of ethics (because of the possibility of misleading students) and pedagogy (since I would like students to distinguish strong and weak arguments for a method being superior to another). I would like to know others' position and arguments on this issue.

I'm also very interested in references that address the pedagogical aspect, based on either research or experience. The only such reference I could find is this blogpost (in Hebrew), written by a teacher, that defines "what's in my pocket" questions to avoid as teachers.

Edit

I might have overdone the abstraction, and made the example trivial. Consider the question

Select the single most appropriate method, out of the list {A, B, C, ...}, to solve each problem in the list {X, Y, Z, ...}.

{A, B, C, ...} are standard methods, all taught in class. Each other problem, {Y, Z, ...}, has a unique method most appropriate for solving it, and the ambiguity rises only for X.

Answer 1

I think the focus on ethics is a bit misguided. If your professor genuinely believes that asking ambiguous, "what's in my pocket"-type questions (a closely related concept is that of a "gotcha" question) on an exam is an appropriate testing strategy and serves a useful pedagogical purpose, then I cannot say that they are behaving unethically.

So the problem as I see it is not that the professor is behaving unethically, but simply that if they believe such a thing then they are wrong. I can't support this with any scientific argument based on research; but just based on my own experience with teaching, one major reason why I'm pretty sure this plan will not work as your professor expects is that students in the US (and other countries I'm familiar with) have been conditioned to expect, over their many years in the educational system, that all exam questions, in STEM and related technical subjects at least, have a single, unambiguously correct, answer (unless explicitly stated otherwise).

Now, a professor might be of the view that this is poor preparation for real life, where situations are typically much more vague and nuanced, and a suboptimal way to develop critical thinking skills. So it's tempting to think "why don't I design an exam that simulates real life more closely, and this way test for more of the things that really matter and encourage students to really think". This is well-intentioned, and perhaps even correct in an idealistic sense of being closer to the Socratic method. If you were gathered together with a small group of students on an island somewhere and had all the time in the world to develop their intellect, and no earthly concerns about schedules, tuition, the need to test and grade hundreds of students in X amount of time with Y resources with no easy chance for a do-over if something goes wrong, etc., then asking such questions would be a wonderful method to stimulate their thinking.

On the other hand, here in the real world, the conditioning I mentioned is so powerful, and the conditions under which students are tested so stressful and unforgiving, that asking questions with more than one correct answer (without stating that explicitly in the question) is a very clear recipe for trouble. A lot of students are already so stressed out and anxious taking tests that it's difficult to them to think creatively or be resourceful in an exam setting. What I expect will happen is that many of the students will obssess over whether A or B is really the correct answer (despite your professor saying they "shouldn't obsess over this minor detail"), will get confused and spend unreasonable amounts of time on this issue, will end up not having enough time for other questions they are perfectly capable of solving, and will end up frustrated and hurt and feeling that they have been treated unfairly over this testing strategy. And of course, they could complain, leading to a nasty headache of a problem for you and the professor to have to explain to the undergraduate program chair.

Here's a final thought to consider: it's hard enough -- by which I mean essentially impossible -- to get students to believe you're treating them fairly when you're bending over backwards not to create problems of this type and "just" giving honest, error-free, clear exam questions each of which tests one unambiguous snippet of knowledge. Even then, some students will still manage to come up with explanations for why they did poorly that somehow put the blame on you and your teaching/testing/grading practices and not on themselves for not studying hard enough. So the way I see it, things being hard enough already, why make them intentionally even harder still? This particular headache is easily avoidable. Let the professor try out their creative teaching ideas during the semester and not in the high-stakes, pressure-cooker environment of an exam. For the exam, my advice would be to stick with standard principles for designing exam questions that follow the same conventions everyone else uses and that students are used to. Anyway, that's my unscientific two cents.

Answer 2

In a "select one" style multiple choice question, the expectation will be that there is only one correct answer. So, if you are planning on accepting either A or B as a correct answer, this should be clearly stated on the question, since it is an unusual (and therefore unexpected) format for a MCQ. If you leave it unstated but accept either answer anyway you will prejudice students who end up confused as to why there seems to be two valid answers and waste time trying to figure out which one to choose.

A better choice is to use the more standard "select two" style question. This is straightforward, clear to the student, and actually a better quality question since you are now testing that the student knows A and B rather than A or B. In your question the question would become: "Select the two most appropriate methods, out of the list {A, B, C, ...}, to solve problem X".

Exams exist to test knowledge, not to be tricky and make the student spend time trying to work out what the question means.

Another way to look at it is: is there any need to have an ambiguous question when there is an unambiguous alternative available? What purpose does it achieve?

Answer 3

I think this would be appropriate in a written exam with no explanations of answers possible only if the professor has made a specific point of this issue. If it is actually ambiguous to the student who paid attention and did the exercises and such, then no.

Multiple choice questions normally have one clearly correct answer and a couple of others that are close to being right, but are not, where the reasons should be clear to the attentive student. But that seems not to be the case if a TA is confused.

Alternatively, and this sometimes is used, it would be fine if there are only small differences but the prof is willing to accept either answer as correct.

Simple multiple choice questions aren't the sorts of things to be used to pick out fine subtleties, especially in a case like this, where there are multiple considerations that people make in choosing "best method...", and context matters.

Answer 4

How about changing the question to "select an appropriate method"? Or perhaps "of the following methods, select one of the appropriate ones." I think that asking for "a single method" is still confusing.

A number of students will absolutely obsess over this detail unless it's made clear that they should not, because they expect a multiple-choice exam to have one correct answer per question. It's fine to change that expectation and have multiple correct answers, so long as you say so.

Answer 5

It sounds like in reality, both A and B are appropriate methods for solving X. If you artificially removed either A or B as options, would you not also be misleading students by guiding them to one specific method for X?

My advice is to accept the ambiguity. Ambiguity exists in reality and there are often multiple correct ways of approaching a problem. If all correct approaches are marked as correct on the exam, then the exam is testing students' preparation for realistic application.

Alternatively, if you want to test students' knowledge of a specific method, ask students to actually solve problem X specifically by using that method.

Answer 6

I don't have a lot to add to existing answers. I agree with the following point mentioned by others: Students are conditioned to not question the questions on exams, especially if they're not free response questions. Exams don't exist for the purpose of making students question the questions. It's expected that exam questions are straightforward. (I suppose the only exceptions are when the instructor has placed particular emphasis on making students question questions and their underlying assumptions.)

In addition, questions that have more or fewer correct answers than implied are considered to have errors, which then need to be announced to everyone as the grading will be different than expected for the question. For example if there are more correct answers, then all of them are marked as correct. If there are fewer correct answers, then the question is thrown out entirely--and in some cases, if the instructor is made aware of this during the exam or in the time between printing the exam and handing it out, then the instructor will announce that the question should be completely ignored. I speak from personal experience from when I was a student and an undergrad TA, but I think these scenarios should be familiar to a lot of people.

Perhaps this personal anecdote will give you ideas for the future: Some of my professors made multiple-choice questions that could have any number of correct answers, from no choices to all choices. These questions specified that as few as none and as many as all of the choices could be right. Grading was based on how many choices we correctly put in the correct "state", namely checked or unchecked since the choices used checkboxes. (These professors are in the same department and use the same online quiz system that belongs to the department.)

Answer 7

I'm not sure if an anecdotal answer is appropriate here. Many years ago, all children in England had to take an exam called the 11-plus (referring to the age group). This was very important in those days because it led to a binary decision about what sort of secondary education you would receive.

I was never coached on how to do well in the exam and walked in a complete novice. Right from the start I saw that many of the questions had justifiable multiple answers - yet only one check box was allowed for each. I spent a lot of time trying to decide how to answer each question and which answer would most likely be considered the "right" one.

I can't remember the exact questions now but let's invent one. What is the next number in the series? 1, 3, 5, 7, ... Now, as any mathematician knows, there is potentially an infinite number of answers to this question. We can guess '9' because it is more 'obvious' but, as a child, I spent a lot time looking at all the proposed answers and finding a justification for each.

Luckily I got a sufficient score to go to the 'elite' type of school. However, I'm sure it was touch and go because of the extra load of trying to choose between what I perceived to be multiple, equally justifiable, possibilities.

The solution

Warn the students! Tell them what they should do if they meet a multi-answer question and have to choose one. Students should be taught strategies, not just rote methods.

Answer 8

Apart from the other answers, your premise is wrong.

The question does not have 2 correct answers, it has 0 correct answers. It is impossible to select the single most appropriate method as there is no "single" most appropriate method.

With that in mind, if the students complain, you will have to accept every answer as correct (or remove the question). Even accepting A and B is not correct as none of those are a correct answer to the question.

So your question shouldn't be if you can ask ambiguous questions. It should be if you can ask questions with no correct answer. The answer is obvious.

The question should be removed.

Answer 9

The sentence:

Select the single most appropriate method, out of the list {A, B, C, ...}, to solve each problem in the list {X, Y, Z, ...}.

has very clear implications to typical English speakers. You could expand the sentence a bit, to make the implication more explicit:

For each problem in the list {X, Y, Z, ...}, there is a single most appropriate method in the list {A, B, C, ...}. Select it.

If you agree that these sentences are equivalent -- that any typical English speaker would read the first as meaning the second -- then I think it's very clear what the issue with the question is. If one of the problems does NOT have a single most-appropriate method in the provided list, then this clear implication of the question is false.

Would it kill your professor to add a short note to the question? "If multiple methods are perfectly and equally appropriate, select any one of them."

Answer 10

It seems that this question is in some ways asking for a subjective answer is but worded as asking for an objective one. As a student I would prefer to see the question phrased more along the lines of

Select a single method you think is (most) appropriate from {A, B, ...} to solve problem X.

If the student has a choice between two similarly appropriate methods then, to me, asking for the single appropriate method seems... strange. Surely if you want them to argue which they think is the most appropriate then asking for that makes more sense.

If there are multiple "most" appropriate, then including wording that asks for the "most" could also be considered unfair (although much less so in my opinion) and could still lead people down a hole of questioning their understanding in a potentially unhelpful way, as others have mentioned. Simply asking for "a single appropriate method" could be "fairer". You suggested that you are asking for many solutions to many problems in the paper, so it may make more sense to include "most" in this context.

Edit: Another option would be something much simpler.

Choose an appropriate method to solve problem X from the following options: {A, B, ...}. [Explain your answer.]

The original wording I suggested was trying to keep as closely to the question in OP, but in practice that feels clunky. I feel that this wording maintains most of the meaning while reading better.

The problem I see with this is that while "Choose an appropriate method" is not ambiguous, it runs the risk of people thinking they have too much of a choice, and assuming that there are multiple potential answers for all of the questions - which you said is not the case for most of the questions in OP. For this reason I would stick with

...a method you think is (most) appropriate...

as it makes it clear that you are looking for a "most appropriate" solution, but that it may be open to interpretation.

Answer 11

Directly to your title question ... Not purposely. We can ask open-ended questions. The two (open-ended and ambiguous) are not the same. It appears that you have a good sense of the responsibilities to be clear in how you are phrasing your questions to the students so that you can be clear to yourself later why you are marking their answers for being incomplete.

Unfortunately, we do ask ambiguous questions accidentally. You have a good sense of the fairest approach. Accept both answers. I would only recommend that you follow up with the students to admit the ambiguity. Subsequently provide a clear statement to the students that presents the correct (unambiguous) question that you should have asked and might perhaps ask in the future.

Finally, I would suggest that even your rephrasing of the question for this case seems questionable. If you really mean to have two answers that have exactly equal merit on their objective arguments alone, then the opening statement must not confuse that more than one answer is a possibility.

You may mean instead to say this:

• Select an appropriate method or methods ...

This works if two (or more) of the methods are appropriate and all others are absolutely not appropriate.

You may mean instead to say this:

• Select the most appropriate method or methods ...

This works if all methods are appropriate but one or more are MORE/MOST appropriate.

Answer 12

If you say "single" but accept two answers you are lying to them, that is unlikely to end well.

Most multi-choice exams begin with the instruction "For each question choose the answer which you think is best"

Given that, I think you should rephrase the question to

which method listed below is appropriate.

They already know you want what they think is best. you also have the option for saying "E: options A and B"