When teaching, which question types are effective for stimulating deep processing of the material in students?

Question

In our university, we have a discussion about different question types to use in a module in our Electronic Learning Environment called 'self-tests'. We use this module to stimulate active processing of learning material by students in an automated fashion, enabling us to offer courses with a high degree of interactivity while remaining feasible for large numbers of students.

Our discussion concerns specifically which question types we need to implement in the Electronic Learning Environment. Some example of question types are the following:

• Multiple choice questions
• Open questions
• Ranking questions
• Images with 'hotspots'
• 'Likert'-type scales and matrix questions ('arrays' in LimeSurvey)
• Checkboxes
• Sentences (or complete paragraphs) where words have to be entered to complete the sentences

Some applied examples would be:

• Please organise these concepts to reflect the developmental stages distinguished by Piaget.
• Which of these types of sampling are appropriate? In this painting, point out where the artists uses [INSERT BRUSH TECHNIQUE HERE - sorry, I'm a psychologist, not an art scientist :-)]
• Given the above material properties, how much load can the bridge bear?

A colleague recently voiced the position that all other question types 'boil down to yes-no decisions' and that therefore, there is practically no variation in the types of information processing students engage in when working with the different question types.

However, in this discussion, nobody has as yet managed to produce any meta-analyses of the evidence, or in fact other forms of empirical, or for that matter, theoretical, evidence. At the same time, I can't imagine this hasn't been studied yet. However, queries in Google Scholar using keywords such as 'information processing', 'cognition', 'formative tests', 'interaction' and 'learning' didn't yield anything. Lots of opinions and examples, but no evidence.

So my question is: does anybody know of literature presenting empirical or theoretical evidence regarding differences in processing between different question types (methods of interaction)?

This is the kind of thing academics (as teachers) should all have readily available I guess - but I'm afraid I don't, and I don't know anybody who would.

I don't have any 'dogs in this fight' - if there is evidence that all question types perform equally, that's fine. I just want evidence, regardless of which way it points :-) As long as the evidence is methodologically and statistically rigorous I'm happy.

One answer would be, of course, "it's not about the question type, it's about what exactly you do with it and how you embed it in the course". However, this ignores the fact that different question types enable/prohibit different problem solving approaches. It would basically just change the question into "assuming the question types are applied properly, which serve which goals best?"

I'd be very grateful for any pointers towards literature where (preferably experimental) evidence on this matter is presented.

[this question is kind of like a more in-depth version of this one, which is about whether online quizzes can work in general]

Answer 1

This is a question which is far too big to reasonably answer, particularly without knowing the subject matter and tasks of importance. In terms of theories with solid evidence, ones that I find useful as guidelines are:

• Chi's Active-Constructive-Interactive framework. However, it sounds like your system does not naturally lend itself to constructive or interactive activities. This is unfortunate, because I've seen studies with evidence of learning gains for things as simple as quasi-interactive tasks like allowing people to continue answering multiple choice questions for partial credit after wrong answers (e.g., wrong answers trigger hints for further reflection).
• 25 Learning Principles: More of a list of known effects rather than a theory, it is still useful to know. The primary issue with some of the items on this list is that it is hard (maybe impossible) to optimize for all of them in designing a course. Also, certain content fits certain approaches better, either because of the type of knowledge or expectations by teachers/students (e.g., embedding math into stories might be pedagogically valid, but jarring/disliked due to unfamiliarity).
• Aligning Media to Cognitive Tasks: While there is a broad literature on this (Mayer obviously being the big US name), Schnotz's work highlights the relevance of picking multimedia that naturally communicates the information you are trying to convey. For example, it is incredibly hard to verbally describe an individual chair (picture worth a thousand words) but equally hard to show enough pictures to imply the simple concept that "most chairs have 4 legs." This intuition implies two things. First, that much of "learning styles" literature represents preferences, and shouldn't be mistaken for what actually leads to better learning (since the content affordances will almost always dominate the best approach to use). Second, it highlights the need to align your learning tasks to the modality that makes the right information salient and accessible, at least initially. Later, if the authentic application of those skills involves tasks where they are not salient or easy accessible, tasks to represent that greater complexity are also useful. This represents one incarnation of the general strategy of scaffolding, where tasks are simplified/supported initially, but increasingly approximate unassisted performance of the cognitive tasks needed. This has implications for question sequencing.
• Multiple Representations: Related to both of the above, I think it is important to note that multiple ways of testing the same information is important. This has a lot of backing behind it, which is why it is big in the US Common Core for math. It is suspected to have two major mechanisms: a) multiple pathways/cues to recall the same information and b) understanding of the abstract relationships/processes rather than brittle task-specific procedures tied to one representation. This would not generally apply to just different input controls, though, which are likely to be a last-step after the desired learning-relevant processing is done. Or, put another way, if your question leads to knowledge that is so brittle they fall on their face because they have a checkbox instead of a multiple-choice, they probably won't be able to apply that knowledge in practice anyway.
• Generative: Also noted in some of the above, and relevant to the question types that you list, open response is different and generally better. This is particularly relevant because many of the above types (e.g., multiple choice) are actually very hard to build correctly, with most teachers using a strategies like: 1 right answer, 1 obviously wrong/off-topic answer, 2 variants of right answer with flaws. They're very prone to test-gaming, unless you start with open responses and build one with a right answer vs. common misconceptions, for example. Gaming behaviors are known to reduce learning, probably because learners spend their time thinking about the strategy to game the answer, rather than actually processing the domain-relevant information.
• Comprehensive Testing: Testing on the full set of knowledge so-far has also been shown to increase learning gains. This was noted by one major report as the single simplest and most effective way to increase course-level learning gains, if I recall, but I don't recall the cite off the top of my head. I also cannot recall if this had implications related to studying more or if it was more of a repeated practice issue.
• Question Taxonomies: There are also a few question taxonomies to look into, which have some indications that deeper questions (e.g., causal reasoning) result in deeper learning. However, deeper learning is not necessarily more learning. The relationship can be quite complex, and if all you really want is just recall (e.g., the end-task is recall) then deep questions may not be efficient for that goal. As always, aligning practice to your target task is important: if your ultimate task will be shallow, shallow practice probably works. There might be some value to "multiple representations of different shallow practice" using MC questions vs. Y/N vs. checkboxes maybe, but that seems like such an uncommon situation that I am genuinely not sure if anyone studies it.

In general, I would say that I don't see tons of difference between any of the closed-response questions (Y/N, MC, Checkboxes, Ranking, Likert). Yes/No is a bit flimsier than the others because you only need to evaluate one assertion, but I'd still say you'd do similar thinking when given the same question regardless of how you're entering an answer. I'm not aware of any research that shows substantial differences between these techniques off the top of my head, and even if differences were observed there's a good chance that they're not useful differences. As with the learning styles literature noted earlier, just because you can find a difference when everything else is kept equal (controlled) does not mean that the difference is useful in practice because you generally have much stronger factors to manipulate. Open response is different because you get less cues (e.g., you can't just use recognition, and instead need to use recall). There is some research indicating that is better, but applying the caveat of what better means (e.g., might be deeper but less efficient, if all you truly need is recall).

I will note that there are a few systems out there for making interactive questions for courses. CMU has the CTAT system. WPI maintains the ASSISTments system. Both of these have various levels of retry/hinting support for interactivity, among other types of adaptation. There are also some similar systems for doing this with open-response formats (i.e., dialogs), but I don't currently know any good ones that can yet be easily and freely embedded by instructors into courseware and that also have professional support staff available.

Finally, if it really is the input controls that you're most interested in, you might find more literature on that in fields like HCI or marketing survey design. Both of those fields tend to focus more on the affordances and optimizations to input mechanisms, while learning sciences and education is typically more interested in the content of the questions (other than the notable concerns about choice-based questions and simple active learning tasks in general). But personally, I think that the big gains aren't in those minor tweaks.

Answer 2

I'm a teacher major with about 2 years of experience. Unfortunately I cannot cite research, but offer some of my own experience with some theoretical background.

According to the constructivistic theory of education, learning is mainly made by an active effort of the learner to make a mental construct. Therefore, and according to my own experience, the best questions are which require an insight to solve.

For example, good puzzles almost always require you to discover a new concept, unless you knew it, of course.

SPOILER ALERT

There is a Sam Loyd puzzle where you have to make a donkey (or horse) out of a few pieces of paper, which never seem to give the right shape together. But when you put the pieces in a circular fashion, you can make the outline of a horse (the negative, i.e. the table you put the puzzle on). So if positive/negative image is what you want to teach, perhaps in the context of Gestalt or graphic design, it's a great way to make them grasp the concept by utilizing it.

SPOILER END

Lateral thinking puzzles usually do this, because when you reach the limit of your knowledge, you need to reorganize your thinking to go forward. And that's what teaching is about in my (and constructivist teacher's) opinion.

Some other ways to do it would be presenting case studies (word problems), when they need to explain what would they do in that situation and why (both can be multiple choice). Or let them make up their own questions about the topic, which is also great for learning -- and you can use the best ones for later.

There's research that suggest that simply reviewing study material is not enough for long-term memory. You need to try and remember the answer before checking it. I don't have links, but Cal Newport's probably write about it.

Another great resource is Made To Stick by Chip and Dan Heath.

So in my opinion, it's not the format of the answer what is really important, but rather how engaging the question is. But maybe there is some research about that, and some formats may be better in a given situation than in others.

Answer 3

Agree with the comment above about Bloom's. I take a different tack though.

Why not ask questions that matter in the particular discipline. I use the motto of "let the subject speak for itself". If your question matters, it is worth working on it.

If it doesn't matter, why waste time trying to 'tickle' a student's brain. They aren't actually dormant. Most students are pretty busy, more so than professors, and they will work on things that matter.

Equally, keep your assessment format constant throughout the course. If it matters, then it matters to practice it. So model what you want done in class (be the role model of the way of thinking and talking in your discipline) and make sure the coursework and exams are all in the same format. Then they understand what you want and get better at it.

Most courses these days are only 10 weeks so there is very little you can actually teach and practice. A good example though is Andrew Ngs course on machine learning (Coursera). Each week he taught a new machine learning technique. He taught the maths on powerpoint and then set a programming assignment that could be finished in one evening. Students submit it online and it is tested against another data set designed to trap the most common errors.

It is memorable and doable. It has a minimum standard of performance in the programming. Students know they have walked through the theory and can go back to it when they need. Elegant, tidy, predictable, repetitive but with each new assignment taking a student upwards in content, upwards in topicality and upwards in having practise core skills once again.

Hope that helps. (PS never over do the teaching. Students aren't daft. Be orderly and let them get on with it. They will).