3

I'm new to academia SE and I'm not sure if this question fits here, but I think this is the best place I can ask. Please let me know if this question doesn't fit here and suggest me better place to ask.

Let's say that I'm a teacher and there is a student. We ASSUME that if a multiple choice question is given, then we can predict the probability for each choice that the student will choose. For example, if a question have 4 choices A, B, C, and D, then we can predict that the student will choose each answers with probabilities 40%, 30%, 20%, 10%. We will always assume that A is the correct answer. Then my claim is the following:

For education, it is better to give a problem with probabilities 50%, 50%, 0%, 0%, than 25%, 25%, 25%, 25%.

The intuition behind this claim is the following: if we give a question with same probabilities for each choices as 25%, then the student might randomly guess, and she may learn less from solving the question. However, when a student faces a problem with probabilities 50%, 50%, 0%, 0%, then she may choose between A and B. Here 0% means that she (almost) surely knows that C, D can't be an answer. If she did wrong (choose B instead of A), then she may learn more from it because she only need to see difference between A and B.

I want to find a suitable reference that justifies my claim, with some real experiments. I tried to search hard but I failed to find it. If anyone knows such reference (papers or whatever), please introduce it to me. Thanks in advance.

Improve this question
4
  • 1
    What do you mean by better? In my view, the probability of a student choosing a certain answer (how would you define this anyway?) matters less than what that choice tells us about the student. For example, you could have Q: Compute (4+2*8)^2, and answers A: 400 (correct), B: 2304 (Student added before multiplying), C: 272 (Student thought (a+b)^2 = a^2+b^2), D: 357 (there is no plausible path to this answer, student is either guessing or wildly confused).
    – Solveit
    Jul 4, 2019 at 2:10
  • @SolveIt I agree with you. Actually, the way I predict the probability uses some estimation of a student's understanding for a question, so I think that the expected probability contains some information about student's understanding for the question.
    – Seewoo Lee
    Jul 4, 2019 at 2:30
  • I don't know where you're going with this (in particular, the part about the test taker "may learn more from it" makes no sense to me; maybe you meant those who interpret the test results instead of the test taker?), but a more nuanced view in which the probabilities are a function of the test taker's overall performance on the test is a well known tool in psychometrics.
    – Dave L Renfro
    Jul 4, 2019 at 3:03
  • With conditional probabilities you can create any structure you want. E.g. make a question where P(choosing answer A | knowledge of facts 1, 2, and 3) is high, while P(choosing answer B| knowledge of facts 1, 2, but not 3) is high. Not only does a mistake tell them they are wrong but also in what aspect of the material. The limit of course is your creativity in thinking of these questions.
    – A Simple Algorithm
    Jul 4, 2019 at 5:33

1 Answer 1

Reset to default
2

You could probably ask this as well on MathEducators. To be clear, I haven't found any academic study about it, but even if there was one, you should know in advance what you are expecting to achieve.

I interpret your answer as in: "Should multiple questions allow options to be eliminated with partial effort?"

The answer of course, is that it depends a lot. Think carefully about:

What kind of test are you preparing? Is this a regular exam for elementary school that accumulates points which eventually determine if students are approved for the next year? If so, and if questions are easy already, I'd go for the equal chance for each alternative.

Or are you preparing an admission exam? In easy admission exams, every bit of knowledge should count, so that the guys who knows nearly nothing is ahead of the one who knows nothing. In difficult/competitive admission exams, people who are to be admitted are limited in their score by their performance accuracy, not by their knowledge, and possibly by their speed. So these tests are designed to have very good students getting around 70% of the maximum score. I've known such tests to contain this kind of problem either to mitigate the effect of very difficult questions (you give an easy way for the student), or to give time to the ones who master the subject well enough to think how to approach the problem even if they know a standard (and time consuming) solution method.

What average score should students get?

Again, If you are expecting bad students to get good grades, maybe you shouldn't bother to make it even easier, but if you expect good students to have bad grades, then why not?

Are you evaluating effort or just results?

Maybe you are giving 3 absurd options and 1 reasonable 1. Maybe you are giving 4 reasonable options, but some of them can be eliminated with knowledge of the subject. The first case only rewards students for reading the answers, while the seconds can measure some level of knowledge on the subject. Makings options impossible to eliminate rewards the student for the effort of solving the question, this is well represented in calculation intensive questions. But if the question is simple and "you either know it or you don't" (think asking who painted the Mona Lisa), you can make eliminating options difficult.

So effortless question does not give much room for allowing options to be eliminated, but work intensive question do. IMHO.

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .