41

There has been a lot of talk about online teaching in relation to covid-19 and campus shutdowns.

This question isn't about that though: my question is about what ways online examinations should be treated in counterpoint to written examinations.

I'm going to be transferring my course's written examination into an online environment. This examination involved 2 hours of students, with a pen and paper, answering questions both short and long in a typical examination environemnt.

An online exam however cannot be policed. Demanding it be anything other than open book is unrealistic. What methods can be deployed such that it is a genuine test of students' knowledge and capabilities?

  • 11
    What's the course about? Math? Science? Programming? Humanities? Or any of them in general? – scaaahu Mar 17 at 12:10
  • 3
    @scaaahu Programming, but very little emphasis on coding in the written exam. – Stumbler Mar 17 at 15:06
  • 3
    Are honor codes a thing in your country/institution? (I am not vlaiming they solve the problem) – user111388 Mar 17 at 16:53
  • 9
    @user111388 Have those ever worked? – Mast Mar 18 at 12:32
  • 19
    My cynical response to your question title is that we don't even know how to do this using normal exams. – xLeitix Mar 18 at 14:49
36

First, a two hour time limit might be difficult to enforce or to guarantee, especially if it uses a real-time clock. At some level of scale you may start to find that some students didn't get the exam paper or were unable to return it by the deadline. So, I'd start by rethinking if a time limit of less than a day is really essential to your exam.

Second, you can pre-vet any questions you ask by doing searching yourself online for possible answers and responses. Reject questions that have too much online presence, or be prepared for getting that back as answers. Presumably the test isn't about effective online searches.

Third, give up the idea of asking students for facts. That is obvious, I'd think, as facts are cheap. But even facts requiring computation, such as in mathematics or statistics can be generated with tools such as MatLab or Mathematica.

The best sort of questions, I think, are those that require either or both of interpretation and insight. Questions about the why of things rather than the what and how are much better. However, these are the hardest to create, the hardest to answer, and are likely to have the widest variation in quality from students. To grade them might require quite a lot of interpretation: does this student show any insight at all into the subject? Pass-fail grading can be considered. Repeat attempts can be considered.

As an alternative to an exam, you can consider individual portfolio development by students. Students write or otherwise create some artifacts according to some criteria. Plagiarism concerns come in to play here, of course.

I'll note that honor codes can help to a certain extent, though provide no guarantees. And honor codes created in the moment are probably less effective than those of long standing.

| improve this answer | |
  • 1
    Why would you suggest that honor codes created in the moment are probable less effective? I would suspect that they would be more effective, with more novelty to them and less chance for them to dull with time. I would suspect that attention and thought to an honor code is what ultimately makes it effective. – Ben I. Mar 18 at 3:13
  • 10
    @BenI., I don't have any real evidence. But when a student applies to, say, Dartmouth, they already learn of the honor code and what is expected of them. There is a history of its application, including violations. Students have participated in it, including the statement of it and its application. A new code might just seem like something imposed. But it is really just a guess about relative effectiveness. – Buffy Mar 18 at 10:53
27

I think a lot of us are dealing with this right now. Here's what I and some of my colleagues are doing.

Open book exams: As a lot of people are saying, there's no practical way to police students' use of outside resources, so don't. Write the exam with the expectation that they are using notes and possibly even Google, and let them know that. You don't want to disadvantage students who think they're being honest!

Interpretive questions: Google is great for facts, but lousy for providing interpretation. As a math instructor, one of my major concerns was the existence of software like Symbolab and Wolfram Alpha that can do most calculations. But there's no software in existence that can solve a decently well-written word problem, or that can explain what particular features of a graph mean in context.

Class-specific questions: One of my colleagues, a history instructor, has the problem that the topics involved in the exam are sufficiently well-studied that interpretations are actually freely available online, and a reasonably clever student would be able to paraphrase well enough to avoid detection. What he's planning to do is to ask the students to relate the topic to the class itself, asking the students to reference in-class discussions in their answers. Because Google has no way to know what was discussed in class, this is pretty robust against cheating.

Image recognition: Google does have reverse-image-search functionality, but it isn't very effective if you made the image yourself. As a result, asking a question that requires students to understand an image can be useful; in my exam, I asked students to supply a function that matched a given graph. It's worth noting that, since images are largely incompatible with screen readers, you might have to be conscious of any disabilities among your students.

Time: If a time limit is important, have the exam available for only slightly longer than that time slot. My exam was two hours long, and I had it available for three hours. That limited the ability for students to communicate questions to each other, and it also allowed some flexibility for students running into technical problems. Personally, if possible, I recommend just writing an untimed exam, written with the expectation that students might be communicating with each other.

Justification: Most students do a very poor job of explaining reasoning that isn't their own -- use that. If an exam problem requires students to justify their answer, it's that much harder for them to get answers from one another.

| improve this answer | |
  • This is a really good answer with useful suggestions! – user2705196 Mar 18 at 22:33
  • +1 for "Interpretive questions": this is the type of most questions I have been getting in interviews – onurcanbektas Mar 19 at 13:19
  • 1
    +1 for taking into account students with disabilities. – jhyatt Mar 19 at 19:12
  • 1
    you are right that there is no software that can solve word problems, but just to alert everyone, I (and many others) have had students snap a pic of a word problem, post it to (for example) chegg.com, and immediately get an answer - and this was for in-class exams that I thought i was decently proctoring. – usr0192 Mar 25 at 6:43
10

I thought this might be useful as I haven't seen it posted yet.

One idea I have seen professors deploy (I am a PhD student so I sometimes get to help think of these things) is to introduce some complication into the test. Some ways I've seen this done (and some ways it's being done to me this semester):

Open book, harder problems

The idea behind this one is to tackle the problem you pointed out directly - how to make a test a test while more-or-less being forced into an open book scenario. In this option, you would simply increase the difficulty of your test appreciably. Perhaps have some questions that, when answered correctly, would confer a C (or equivalent) grade. For the other questions make them incrementally harder. The idea here is to ensure the student is learning something. One way to do that is to push them modestly outside their comfort zone with questions that can't be simply looked up in their text. My program is Computer Science. One way this could be done in my field is to, instead of requiring a student to demonstrate the steps of an algorithm, prove something small. Maybe an example of proving an invariant of something, or a challenging application to something they learned in class. Enforcing a time constraint would be very difficult for someone unless you had a proctor for each individual.

Open book, research

Ask the students to interpret something they've learned in class in perhaps a new light. Depending on the course level you are teaching this could be something very basic all the way to something a PhD student might be expected to do in their research (obviously it must be attainable and reasonable - but I think you get the idea). Again, unfortunately enforcing a time constraint without a proctor would be difficult.

Both of them require more work from you unfortunately. At my level I have not seen any timed online tests mostly because the infrastructure just isn't available to be spun up quickly for classes that weren't already doing it. Additionally, the material is quite complicated and doesn't lend itself well to chunking out into online questions that a computer could grade.

Good luck!

| improve this answer | |
8

I’m considering making my final exams oral via Skype/Zoom/WebEx/Teams.

My reasoning is that, for the size of class I teach (20 to 24) and the usual exam duration and marking time, it will take me about the same amount of time.

I believe that I can usually ascertain a student’s grade to A/B/C/D/F within about five minutes of questioning. The finer grades will take a little more time, but again still within the time I usually spend on grading and proctoring.

I have a midterm coming up, and will try it out there.

Advantages: you’re pretty sure it’s the right student. They don’t have time to look much up. And any in-room help will be obvious.

Disadvantages: scheduling it may be difficult. Students are not used to it. Getting them to do computations / calculations may be problematic.

—-

Edit:

So far, I’ve done about about one hundred oral exams and it’s worked very well. The scheduling issue has a tech solution: I’ve used Microsoft Bookings which has direct access to my calendar and let’s the students manage their booking. Plus, I get to schedule when they can book their exams.

| improve this answer | |
  • 1
    I think it is a good idea, with two problems: it may make people more nervous, and with overloaded servers, there may be problems. I hope you find ways to mitigate both. – Davidmh Mar 18 at 16:33
  • @Davidmh Thanks for your thoughts. Yes, when I've done oral exams in person before, I've seen grown men cry in front of a white board because it made them so nervous. Definitely something to avoid. – Peter K. Mar 18 at 17:44
  • 4
    When I was doing exams on paper I would jump and switch between problems to unblock myself. It is not easy to do on an oral exam. Maybe you could give them a list of questions up front, and let them choose the order? – Davidmh Mar 18 at 18:45
  • 2
    @Davidmh Yes, I certainly intend to give them the list of "starter" questions: where I'll start with the questions. Previously, with in-person orals, we issued a (long) list of possible questions that the students could be asked -- before the exam, so they knew what they were letting themselves in for. I'm also amenable to switching between problems if the student gets flustered. It'll be a new experience for them (and me, online), so I'll have to be flexible. – Peter K. Mar 19 at 18:00
  • 1
    Additionally, we have to have an observer ("Beisitzer") in Germany for oral exams to ensure that no one is unjustly evaluated. – Debora Weber-Wulff Apr 28 at 8:26
3

I would resist the COMMON ADVICE to go for some harder, research-y, project-y type questions. They are not as pedagogically helpful for a neophyte. (Less progressive, less basic skills testing.) Instead I would decide that some loss in cheating is worth the cost, versus a cost in learning.

All that said, the following actions can be taken to mitigate* the cheating danger:

  1. Make the exam timed, AT a specific time (this is a good idea in general to exert some discipline...CLAIM your regularly scheduled time), with typed answers submitted in last few minutes of the test. NOTE: you don't need some fancy system. It can be as simple as an email which they "reply" to by adding answers.

  2. Make the tests LESS high stakes by doing more frequent AND shorter exams. A weekly Friday hour long exam is not unreasonable for a 5 day per week class. An every two weeks hour long exam is reasonable for a 3 day per week class. Several studies have shown that lower stakes, more frequent tests are less likely to be cheated on.

  3. Add a signature block (can be typed name) that no assistance was received. The intention is not to stop ALL cheating. But some fraction of it.

  4. Make the exam 10% shorter based on the time for the kids to access the email and reply to it. (And let them know this.)

  5. Make it a little easier than normal. The benefit in driving some feeling of success and lessening the "need to cheat" feeling is worth the small loss in challenging the best students. But if you want, you can do an extra credit (with the clear communication to the class, that it will be significantly more difficult and not to attempt it until all other problems have been attempted. Make the stakes relatively LOW (points awarded) for the EC, so it really is just for the kids that are "acing" the easied-up material.

  6. Make all the questions very clear and more mechanical. Avoid problem formulations that require explanation (possible, but more difficult/distracting to communicate a clarification with students not in the room). No tricky words. Use the symbols they are used to, etc. In mathematics or chemistry or physics, stick to problems that mimic the examples.

  7. For chem/math/physics, it is unreasonable to ask for writing in LATEX or the like. So I would eschew the normal problem style where you check the work and instead ask for actual answers (grade only those, only require the typing of those). But you can compensate by giving more (but more simple) problems or by asking for answers to intermediate steps. For truly essay style questions (English, history, biology), this is less a concern. Just let the kids type (they all type nowadays...but typing an equilibrium problem in chem is not trivial under exam conditions).

  8. Probably some things like diagram drawing (geology, biology, etc.) will be more difficult and I would eschew these questions during the "situation". But if you want, you can do the work to devise questions (e.g. give the word that corresponds to A, B, C, D on a diagram.)

  9. (For 7 and 8, the rationale is not to expect them to have a scanner or to challenge yourself with grading their smartphone snapped pictures. That will be a nightmare.)


Bottom line: 3/4 of a loaf WELL EXECUTED is better than a whole loaf that falls in the water.

*"Mitigate" means partially eliminate; NOT totally eliminate. SE Q&A pedants, please note.

| improve this answer | |
  • 1
    This answer is not appropriate for tertiary education. Basic skills are not the area of interest. – Anonymous Physicist May 15 at 1:39
1

A minor suggestion if you're worried about cheating and this is about objectively graded subjects (as opposed to writing a 2-page essay on Machiavellian philosophy).

With minor programming-fu, send each person a test with a random subset of problems from a collection, and every time a problem goes out, change the value and name of some of the variables. You might jest, but my Dad, who taught college math, handed out 2 different paper exams. A kid got a zero on a problem in versionA, and went to my Dad and said "why a zero? the guy next to me had the same answer and it was marked correct."

| improve this answer | |
  • I did a small test with two versions - simple multiple choice and one student copied the answers from anothers paper same questions but answers in a different order... They copied A for A, B for B and were upset they got zero... If only they had checked which A referred to etc. So students still do it... – Solar Mike May 15 at 10:22
0

I face the same issue.

One point I am considering is time , if you write the questions such that if they answer based on memory then you get a (relatively) structured answer.

But if they spend most of the time shuffling through books then the answer will tend to be “chunks” and “lumps” that don’t perhaps flow as well.

So take care with the amount of time...

| improve this answer | |
  • The absolute worst I ever did on an exam (in class) was an open book exam. But part of the reason was that I stayed up most of the night cramming for the exam. I vividly remember paging back and forth through my physics book looking for answers and finding nothing of use. – Buffy Mar 17 at 12:39
  • @Buffy I discussed this years ago with a professor of mine. He was told (ordered) by the external examiner to make the exam open book (for a thermodynamics / power plant analysis course). So he did, based on the course material. The EE could not do it and had to ask for the solutions::: :) back to the original exam then. The prof did test it on his students though and it was actually ok... – Solar Mike Mar 17 at 12:43
  • 3
    I honestly can't tell if you're recommending short time limits or not. "If you write questions such that they answer based on memory" sounds like an anti-recommendation. – JeffE Mar 17 at 20:53
  • 3
    I disagree with forcing stricter time limits on students. This leads to situations where students cram in facts that they'll quickly forget, which seems like a pointless exercise. Ideally educators should provide students with skills that they'll have going forward, and a student's course grade ought to reflect their success in having acquired these long-term skills. Any mode of scoring students in a manner that reflects a transient happenstance, e.g. that they've crammed a bunch of information that'll soon be forgotten, is mere theater. – Nat Mar 18 at 0:07
  • 1
    @SolarMike OTOH, those "under time pressure" decisions are usually wrong – Carl Witthoft Mar 20 at 15:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.