33

To clarify, I am asking if it's professional to ask questions that while relevant to the subject/course, and are related to the topic, but have not been discussed in class, assigned as homework, reading, etc. and are also not related to any prerequisite class that the students should already know. I am also not asking about "gotcha" questions where it's a quick "know it or not" fact, but rather an entire procedure, proof, or concept of some sort.

For example in class, using a formula from the textbook to solve problems, but on the exam asking for a proof of the formula that has been used.

Another example - in a foreign language class, asking about a never-before-seen word that may be related to some other, studied words or that has a similar sound/meaning in the native language.

Do such questions "make students think out of the box", or are they more likely to discourage students when they struggle with them? While in the real world there may be many problems similar to this where the problem is completely new, is this something that should be taught in an unrelated class?

Does it matter if the test in question will be graded on a curve where even if a student doesn't answer the question correctly at all, they could still get a good grade?

P.S. - this question was prompted by the comments in this answer

  • 14
    Since I'm the one who prompted the question, I'll surely give a detailed answer later on, but let me firstly comment on the adjective "professional". The professionality is not really related to administering "surprise" questions, it's only related to the way you ask them and with which aim. It's unprofessional to ask "surprise" questions with the only purpose of failing as many students as possible. Of course, I consider the way I ask surprise questions as professional, and I'll try to expand on this with an answer. – Massimo Ortolano Nov 6 '14 at 7:54
  • 14
    At Edinburgh University Physics, at least 20% of the exam has to be unseen material; but the pass grade is 40%. Some put higher percentages, though. – Davidmh Nov 6 '14 at 10:13
  • 36
    I consider all exams that do not have this property -- i.e. that they require the examinee to display independent, creative thought based on the course material -- broken. Checking only memorisation and calculation skills is beneath what (technical) university degrees (should) stand for. That is, a professional professor should ask such questions (in order to save the world from rote-learning "engineers")! – Raphael Nov 6 '14 at 15:33
  • 22
    Coming to class drunk, making offensive remarks, forgetting to grade exams etc would be unprofessional behavior. Are you sure that professionalism is the actual problem you have with this? – user9646 Nov 6 '14 at 16:18
  • 3
    Just a thought on the meaning of the word surprise: I'd distinguish between testing insight and ability to transfer and apply the knowledge and methods learned in the course (which covers your example, but which I'd expect in an exam) and suprises like asking about biology on the mars in beginner level biology course (assuming astro biology wasn't in teh curriculum, would require detail knowledge about totally other subjects) or even "give an alternative to the proof we had in the last lecture" (= unprofessional attempt to punish students for absence) – cbeleites supports Monica Nov 6 '14 at 16:30

14 Answers 14

3

Among the answers given thus far, the one which is closest to my thinking is that of xLeitix, and the side note about the context applies too.

Now,

Is it professional for a professor to ask “surprise” questions on a test?

It can be professional. As I said in a comment, it's unprofessional to ask such questions with the only purpose of failing as many students as possible.

Moreover, I don't like very much the adjective surprise in the title: a surprise is something unexpected, but if a professor clearly warns the students that at the exam they will find problems which have not been solved during the course, there is no surprise. So, in the following, I will talk about "new" problems.

Exams and tests have, as their main goal, that of assessing how well students master/understand a (small portion of a) certain subject. As others have well explained, new questions or problems can give a hint on how deep this understanding is.

But apart from the above stated main goal, exams and tests might also have secondary goals:

  • An exam can be an occasion for learning new stuff. The well defined separation between learning and verification, which typically happens in a course, is something that drastically comes to an end when one starts working, even in academia. Learning and verification in everyday life are really interleaved, and many times learning has to be done along a stressful verification. So, a new problem during an exam can be an occasion to continue learning in a more "unprotected" way.
  • An exam can be a hint, one of the many, that what has been taught during the lessons is not the whole story, and that beyond the lessons there is much more: new problems surely deliver this message.
  • For a professor, an exam is an occasion to fish for good students to whom propose a thesis. Giving new problems can be a way to find students who are capable of independent thinking.

So, I tend to give new problems at the exams keeping in mind the above points.

To avoid being too general, let's make an example related to my experience. A few years ago I taught a course about sensors, transducers and signal conditioning circuits for graduate electronic engineers. The written part of the exam consisted in one problems about designing or analyzing a signal conditioning circuit or about evaluating the measurement uncertainty of certain transducer. Due to the vastness of the topic, the course could neither describe all kind of sensors and transducers, nor all possible signal conditioning circuits. So, I decided that every exam would have been made of a new problem, where "new" meant:

  • A problem about a transducer not described in the course. Indeed the exam text contained a short description of this kind of transducer.
  • A problem about the analysis and/or design of signal conditioning circuit not described in the course. The students, being electronic engineers, were expected to know how to analyze electronic circuits, even of moderate complexity. In more difficult cases, hints were provided.
  • A problem about a known transducer applied in an unknown way.

Exams were open books and students could bring the solutions of all previous exams and all the class notes. After the written part, if successful, there was an oral examination which was more about class material.

What was the outcome of this kind of exam? The course was in general very well received by the students, even if the exam was considered hard: the percentage of success was around 30% (the pass grade is 60%). The major complaint was about the number of exercises solved during the classes, but this happens in all kind of courses. My answer to this complaint was that there were, indeed, time constraints that prevented us to solve more problems but, anyway, whatever the number of problems solved during the course, at the exam they would have found a new one (sometimes students ask for more solved problems in the hope that these will exhaust all possible cases).

From this and other experiences along 15 years, I think that students can withstand new problems at the exams as long as the motivations are well explained and, especially, as long as the course is worth of it.

50

I don't see a reasonable answer beyond "it depends." It depends on the question: some surprise questions are not actually so difficult and could reasonably be asked on a test with no special preparation, while others are very difficult indeed. It also depends on the students: you can demand more of experienced and talented students than you can of typical beginners.

There's certainly no rule that says you can only ask test questions that are similar to previous questions the students have seen. Sometimes asking unusual questions can be an excellent way to judge how well students have mastered the material. At the same time, test questions that are too unfamiliar or difficult can be unproductive. This is a balancing act that can be solved in many different ways, depending on the style of the person writing the test.

  • What would you suggest as the boundary of "too difficult" - would it be when over half the class does not get the concept? how would you handle a situation where a student who got it wrong comes up to you after the test and says you never taught it (well enough)? – user2813274 Nov 6 '14 at 4:07
  • 31
    @user2813274: both "too difficult" and "never taught it" are very relative concepts. As an extextreme example, I've had a student complain that my midterm was nothing like the assignments, when that particular midterm had 3 out of 5 questions taken verbatim from the assignments. – Martin Argerami Nov 6 '14 at 5:17
  • 19
    Some of the best questions in my undergrad math courses were 'surprise' questions in that they could be answered in a few lines by people who truly understood the concepts and the theorems, or in a few pages by those who didn't. I was usually in the latter category, but I think this is a good way to go. – sapi Nov 6 '14 at 9:01
  • 2
    If a professor chooses to do so, it can be helpful to explain before the test that a question or two will go beyond the mere facts discussed in class and expect an application. Good tests have question(s) to distinguish A students from A+ students. However these questions need to be prepared very carefully. Has the student seen all the relevant theorems they will need? Consider trying it out in quizzes or as a bonus question first. – nickalh Nov 7 '14 at 11:28
41

Every question on a test should be about the material in the course. Many times, however, the professor may be trying to teach a deeper concept than some of the students have learned. This is what creates a "surprise" question: the professor asks something that requires mastery of the material or insight into its deeper meaning, and the student has only learned material to a relatively shallow level.

For example, when I was a TA for a large undergraduate artificial intelligence class, the class taught two things simultaneously. The underlying concept threading through the whole class was how to think about data representation and problem decomposition. As part of teaching this, the students were also taught a number of standard AI algorithms. The tests then typically involved variant algorithms the students had never seen before. Weak students, who had learned the standard algorithms but not the underlying concept would often do badly and complain about the "surprise" questions, since they were being asked about an algorithm that they had never seen before. Strong students, who were learning the underlying concept, had no problems.

In general, then, encountering a "surprise" question means that the student is failing to learn the deeper concepts that the professor is trying to convey. Where the pedagogical problems lies, the professor or the students, is a completely different question...

  • 13
    So in short, you're saying "learn the subject, not the material" - which I hope most people in education and academia would agree with! – G. Bach Nov 6 '14 at 15:40
  • 13
    @G.Bach Also, "teach the subject, not the material": pedagogy can fail from the professor's side as well... – jakebeal Nov 6 '14 at 17:04
  • 9
    Looking at it from a student's point of view: The homework is the student's opportunity to verify that they have a sufficiently deep understanding of the material. If the exam contains "surprise" questions that require a deeper understanding than the homework did, then how is the student supposed to realise that they need additional study, or at least more focussed study? – mhwombat Nov 6 '14 at 17:26
  • 9
    @mhwombat The homework may well have pushed for deeper understanding without this fact ever being noticed by a student who is middling along on partial credit and help from friends... – jakebeal Nov 6 '14 at 17:58
  • 2
    I almost failed my first degree by arrogantly "learning the subject" only. During my masters, I thought "to hell with it" and gamed the system as much as possibly, getting >90% in several assignments and finishing considerably better than my first degree. There was one professor who I really hated (and still do) but I have a lot of respect for him as 60% of his exam was split between two questions which were each basically "here's an equation, solve it". No amount of studying past papers and memorizing assignment formats would help gamers with that question! – Mark K Cowan Nov 9 '14 at 16:49
24

I guess it depends on what you call a "surprise question". Usually, when you design tests, you don't want all questions to be the same difficulty. Rather, you would want to have a number of basic questions to find out who actually did not "get" the fundamental messages (and should hence fail), some intermediary questions which the majority of students will be able to do if they studied, and a small number of challenging problems, which are there to separate the excellent from the good students.

In my tests, "surprise" questions often form the "challenging" part of the test. I write them in the full expectation that only 10% to 20% of the class will be able to do them, but that is ok - not the entire class should have the best grade anyway. This way, me and others know after the course who the students that really understood the material were, and who just studied a lot.

Sidenote: I teach in a european country where it is usual to have a Gaussian distribution over the entire grading spectrum - it is not like in the US where having a "B" is often already considered a bad grade. Also, at least in undergrad courses, it is not uncommon that more students fail than have the best grade.

What makes "surprise" questions difficult for some students and attractive for many teachers is that they actually test understanding, transfer skills, and the ability to apply knowledge as opposed to mechanical memorisation of pre-learned procedures. This is easy to see in your "formula" example. A student that studied can apply the formula (he knows how it works, and how to apply it, and under which conditions), but only a student who really grasps the math behind it can do a proof that they had not covered before.

  • 7
    This is roughly what I felt about the example given: that a mathematician by definition can construct proofs without seeing them first, and so it's legitimate in a mathematics course to use that ability to distinguish the top. If all you do is repeat proofs you've seen that's very modest mathematics. That said, I did one first-year course where we explicitly used the Divergence Theorem without proof on the basis that the proof was too difficult for the scope of that course, and I would have been most perturbed in that situation to find the exam asking for a proof ;-) – Steve Jessop Nov 6 '14 at 14:28
  • 1
    FWIW it's not true everywhere in the US that a B is considered a bad grade. In many of the (undergraduate) physics courses I've taken and TA'd, it was standard that grades would form a distribution centered on B or C. Doesn't stop the students from complaining about it though. – David Z Nov 7 '14 at 12:10
  • I have a similar method (in math at a U.S. school). For an undergraduate exam, many questions are relatively routine, but 10 to 20 percent may require deeper mastery of the material. In my mind, this is what distinguishes an 'A' from lower grades: demonstrating an ability to go slightly beyond what they have already seen. Of course, for freshman I don't expect as much as for senior math majors. – Oswald Veblen Nov 7 '14 at 12:33
16

In my view, a fair examination question draws on any or all of the following:

  1. Material discussed or presented during class contact time.
  2. Material from any of the items on the course reading list.
  3. Core material from any prerequisite courses.
  4. Material that might properly be regarded as common knowledge for students at this stage in their education (basic mathematics, basic use of the English Language, etc.).
  5. Knowledge that can reasonably be derived as a logical consequence of numbers 1.–4. Here, 'reasonable' is calibrated to the level of the course. Much more should be expected of graduate students who are essentially training to do point 5. for the rest of their professional lives.

In my experience, "surprise" questions usually come about because (a) students have not fulfilled their obligation to apprise themselves of the material in 1. and 2., or because (b) students are not sufficiently capable/comfortable with the subject matter to conduct the logical deductions in 5.

In either case, my view is that it is professional to ask questions that draw on all of 1.—5. In their capacity as educators, the main professional responsibilities of university teachers are to decide upon and deliver the appropriate material clearly, and to administer assessments capable of identifying students' success in mastering this material. A question that does not 'surprise' the majority of students can only test this mastery to a limited extent because it leaves little way to differentiate students who have truly mastered the subject from those who have merely done a good job of rote memorization. I would therefore view 'surprise questions' as an essential tool in professors' fulfillment of their professional responsibility as teachers.

  • 4
    +1 for point 5. Questions that require the student to combine concepts that they should know in order to solve a problem are great. However, questions that require previous knowledge that has not been covered in class to answer correctly are a no-no. Making students do critical thinking is great. Testing whether they've memorized things that you haven't asked them to memorize, not so much. – reirab Nov 7 '14 at 15:59
9

My field is mathematics. I always tried to ask at least one question that looked quite different from anything that the students were certain to have seen before (though of course it relied on the relevant material), or that required them to combine several ideas that they might not previously have had to combine. On an exam in first-year calculus I’d have had at most a couple of questions of this type; on an exam in the more theoretical courses and in liberal arts mathematics courses I generally had quite a few such questions alongside the more routine ones, covering a range of difficulty. All questions, of course, required the students to write something, be it a proof, an explanation, or merely a routine calculation, and partial credit was always available.

I should point out that I was not grading to any pre-set scale. I have always preferred to construct the exam that I wanted and then interpret the results. Indeed, I refused to assign letter grades to individual exams, preferring to reserve that painful chore for the end of the course when I had as much data as I was going to get. Needless to say, I always explained all of this at the beginning of the course and again before the first exam. I also made it clear that I did not have the expectations to which most American students are used to being held: it generally worked out that the A students (apart from the rare curve-breaker) averaged 80-85% over the entire term — and I was not especially generous with A grades. A 50% average was generally a solid C.

6

All my real analysis tests (me being a student) were 50%+ completely new theorems to prove. It was to be expected that one would have to think outside the box to even pass the test. And I would say I learned an order of magnitude more in that class than I have in any with more standard tests. But that's a senior level course and being good at writing proofs from scratch was a skill that had been taught gradually over many lower level courses.

One could argue that being able to pass general tests with questions that require uncovered to vaguely-touched-on material or methods that involve combining techniques in ways unseen in class is the culmination of education. If that's in fact the case, it would make sense to introduce it early to cultivate the skill of being able synthesize new answers from covered material.

But it is important that it be reasonable for the student to know the prerequisite material to synthesize the answer. Don't ask a measure theory question on the first test in real analysis. Ask a question that requires use of the least upper bound property of the reals in a tricky way, for example.

  • 1
    This is very common for senior-level or graduate-level mathematics. It would be silly (and perhaps insulting) to ask students at this level to simply memorize proofs and replicate them on exams. The real goal in math education at that level is to learn background material while also developing the problem solving ability that is vital for research. – Oswald Veblen Nov 7 '14 at 12:37
5

I prefer a system where the homework problems are the most challenging. Also, I'm not in favor of a system where the homework is graded, because this makes it more difficult to choose good homework problems. The idea is that students learn best when they struggle a lot to solve difficult problems. One then has to accept that students may not have been able to do well on a particular problem, even if they are one of the best. Graded or not, homework should still be submitted and records should be kept about the student's performance.

The exam should serve only as a basic test that all the students who have seriously followed the course should easily pass. There is no way you can challenge the students in an exam that only lasts for a few hours as you can challenge them with homework that they would need to work on for several days.

The exam should be judged in combination with the homework. Each student's homework record (graded or not) should be taken into account when judging the exam. If it is found that the homework record is inconsistent with the exam performance, then the student should be invited to speak to the Prof. about the subject. It can be the case that the student was nervous and didn't see the solution to simple problems, such issues can be corrected in an oral examination especially if the student does not know that the meeting is in fact a secret oral examination.

It can also be the case that the student did not know much about the subject and just copied the homework assignments from other students.That will then become clear after speaking to the student, the student will then be given a failing grade for the subject.

  • 1
    The real test, your first assignment at your first job, in my experience is often much harder than any homework you're likely to have done. So why shouldn't a test that decides weather someone gets an engineering degree simulate at least part of this? – slebetman Nov 7 '14 at 4:13
  • 1
    Yes, but this mainly happens in the usual setting where the homework exercises are not that challenging and you have a normal exam. If you consider giving a course where students have to master doing the really technical research level stuff, then the usual system doesn't work well. In fact, you should expose the students to something similar as that "assignment at your first job" that you mention. That's best done with elaborate homework assignments and note that you don't need to give each student the same homework assignment. – Count Iblis Nov 7 '14 at 20:30
5

Teachers have been asking questions that are not know to students for as long as teaching has been around. How else can a teacher make the student step out of their comfort zone where the answers have been clearly laid out and instead help them broaden their mental abilities. It is important that the questions is related to the subject being taught, but if I ask you a question about the really world cases rather than the hypothetical it will require some out of the box thinking. Yes it is professional. It is not if they are questions that the class has not prepared the student to answer.

2

There's no problem with Surprise questions as long as they're a small part of the marks and you tell students that there may be such questions.

Some of my professors would structure tests along the lines of 40% basics, 40% intermediate, 10% hard, 10% Surprise.

if you covered and understood all the basics you could reliably scrape a pass.

If you'd covered and understood all the material well you could get a good mark.

If you'd gone above and beyond and mastered the material on the course and had good general knowledge in the area you could get an excellent or perfect mark.

I liked the system as it meant in later years I could generally drag students I was tutoring through the exam based on the predictable stuff and you were also rewarded for independent study.

2

I don't think this has anything to do with being professional, but to what extent it is fair and/or desirable so ask such questions.

To me, the utility of unexpected questions that completely surprise students also depends on what kind of grading system they are used under, and this has not been addressed in the other answers. Under curve-grading/norm-referenced tests, one of the points of exams is to differentiate between students, so difficult surprise questions can be useful to e.g. test if students have gained a deeper understanding of a topic. Under such a system, it is reasonable (and to some extend desirable) that only a small proportion of students can answer some questions.

Under a criterion-referenced/goal oriented grading system, students are ideally supposed to know exactly what knowledge is needed to achieve a particular grade. Totally unexpected questions might be more problematic here under such grading criteria. However, what is an unexpected question is also subjective to some extent, and the learning criteria could also specifically mention fundamental understanding and the ability to apply the material to new situations. Even so, if a large proportion of students fail to understand or answer harder surprise questions this can partially be seen as a failure of the teacher/course (which not necessarily the case under curve grading), since this could indicate that the teachers have failed to convey either the required knowledge or the learning-goals needed to achive a particular grade (alternative explanations would be e.g. high goal standards or uninterested, lazy or weak students).

1

My answer is a bit biased in favour of students since I had a bad experience with such questions.

TL;DR; 'Surprise' questions are good and necessary in some cases but don't make them weight 50% of exam.

While I was a student we had a couple of professors who would start giving you surprise questions when you do relatively good in your examination (oral). Then they could find something you don't know and seriously decrease your overall result or even fail you.

There were some professors who have made surprise questions in written exams grading them as 40-50% of the test itself.

I personally hate this. In my case it had led to the situation when you try not to learn general course material but to anticipate surprise questions. Student have more than one course at the same time and sometimes there is not enough time (or interest) to have deep(out of program) understanding of all courses and you just want to pass the course with 75% grade or whatsoever.

It of course depends on the field of study. If it is something like theoretical physics it is necessary to have skills to think outside of the box.

I liked an approach of one of my school teachers. You could get 110-115% on the exam and the grade you receive is based on 100%. 90% you could get by regular questions and 20-25% by 'surprise' questions. So if you studied diligently you could have a good grade and if you have spent more time on the subject you could even cover minor issues with 'surprise' questions.

1

I guess it depends on the details, the country (different education culture) and subject (some of them are fact heavy, some of them are more problem solving focused).

I happened to come from a place where a student who can solve only problems in STEM field that explicitly showed in lecture or assigned as reading, considered to be rather mediocre (C level). I assume there are different educational approaches, too.

On the other hand, giving a question way out of blue can be off limit and just mean.

1

I see this problem spanning several levels of information:

On the bottom, we have course information that is memorized. Students should not be asked to reproduce definitions, facts, axioms, and other types of foundation information that have not already appeared in course material.

Next, we have "techniques." Especially in mathematics, we learn techniques for approaching hard problems. In other fields, techniques manifest as the methods we use to make inference, the types of reasoning that we use to interpret new situations. These are very general (like integration by parts, the epsilon over n trick from early analysis courses, or less rigid logic like the the broad historical notion that starvation corresponds with instability which makes revolution more likely) and can be combined in many interesting ways. Plenty of good, unfamiliar problems can be written based on familiar techniques. It can be very appropriate to ask students to develop a new technique to solve a problem on a test, as long as the intuitive leap is somewhat reasonable. This is a judgement call on the part of the professor that can reflect professionalism.

The ability of students to adapt new techniques generally depends on their grasp of broad overarching concepts. Development of entirely new concepts probably does not belong on tests because students are not likely to retain them very well under exam pressure. In general, exams are a method for gauging students' current knowledge/understanding of course material. I would say that it is more equitable and appropriate to introduce entirely new concepts on homework and in lecture, so that students have a greater opportunity to internalize them.

  • What is the problem with introducing a new definition in an exam? This can be a great way to test a students understanding of the material. – Tobias Kildetoft Nov 7 '14 at 8:45
  • 1
    I should clarify. I mean that students should not be asked to define a term that is new to the course. It does make perfect sense to introduce a new definition and then use that definition as part of a problem. – user2127595 Nov 7 '14 at 15:59

protected by eykanal Nov 7 '14 at 1:16

Thank you for your interest in this question. Because it has attracted low-quality or spam answers that had to be removed, posting an answer now requires 10 reputation on this site (the association bonus does not count).

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.