What methods can be used in online exams to genuinely test the students' knowledge and capabilities?

Question

There has been a lot of talk about online teaching in relation to covid-19 and campus shutdowns.

This question isn't about that though: my question is about what ways online examinations should be treated in counterpoint to written examinations.

I'm going to be transferring my course's written examination into an online environment. This examination involved 2 hours of students, with a pen and paper, answering questions both short and long in a typical examination environemnt.

An online exam however cannot be policed. Demanding it be anything other than open book is unrealistic. What methods can be deployed such that it is a genuine test of students' knowledge and capabilities?

Answer 1

First, a two hour time limit might be difficult to enforce or to guarantee, especially if it uses a real-time clock. At some level of scale you may start to find that some students didn't get the exam paper or were unable to return it by the deadline. So, I'd start by rethinking if a time limit of less than a day is really essential to your exam.

Second, you can pre-vet any questions you ask by doing searching yourself online for possible answers and responses. Reject questions that have too much online presence, or be prepared for getting that back as answers. Presumably the test isn't about effective online searches.

Third, give up the idea of asking students for facts. That is obvious, I'd think, as facts are cheap. But even facts requiring computation, such as in mathematics or statistics can be generated with tools such as MatLab or Mathematica.

The best sort of questions, I think, are those that require either or both of interpretation and insight. Questions about the why of things rather than the what and how are much better. However, these are the hardest to create, the hardest to answer, and are likely to have the widest variation in quality from students. To grade them might require quite a lot of interpretation: does this student show any insight at all into the subject? Pass-fail grading can be considered. Repeat attempts can be considered.

As an alternative to an exam, you can consider individual portfolio development by students. Students write or otherwise create some artifacts according to some criteria. Plagiarism concerns come in to play here, of course.

I'll note that honor codes can help to a certain extent, though provide no guarantees. And honor codes created in the moment are probably less effective than those of long standing.

Answer 2

I think a lot of us are dealing with this right now. Here's what I and some of my colleagues are doing.

Open book exams: As a lot of people are saying, there's no practical way to police students' use of outside resources, so don't. Write the exam with the expectation that they are using notes and possibly even Google, and let them know that. You don't want to disadvantage students who think they're being honest!

Interpretive questions: Google is great for facts, but lousy for providing interpretation. As a math instructor, one of my major concerns was the existence of software like Symbolab and Wolfram Alpha that can do most calculations. But there's no software in existence that can solve a decently well-written word problem, or that can explain what particular features of a graph mean in context.

Class-specific questions: One of my colleagues, a history instructor, has the problem that the topics involved in the exam are sufficiently well-studied that interpretations are actually freely available online, and a reasonably clever student would be able to paraphrase well enough to avoid detection. What he's planning to do is to ask the students to relate the topic to the class itself, asking the students to reference in-class discussions in their answers. Because Google has no way to know what was discussed in class, this is pretty robust against cheating.

Image recognition: Google does have reverse-image-search functionality, but it isn't very effective if you made the image yourself. As a result, asking a question that requires students to understand an image can be useful; in my exam, I asked students to supply a function that matched a given graph. It's worth noting that, since images are largely incompatible with screen readers, you might have to be conscious of any disabilities among your students.

Time: If a time limit is important, have the exam available for only slightly longer than that time slot. My exam was two hours long, and I had it available for three hours. That limited the ability for students to communicate questions to each other, and it also allowed some flexibility for students running into technical problems. Personally, if possible, I recommend just writing an untimed exam, written with the expectation that students might be communicating with each other.

Justification: Most students do a very poor job of explaining reasoning that isn't their own -- use that. If an exam problem requires students to justify their answer, it's that much harder for them to get answers from one another.

Answer 3

I thought this might be useful as I haven't seen it posted yet.

One idea I have seen professors deploy (I am a PhD student so I sometimes get to help think of these things) is to introduce some complication into the test. Some ways I've seen this done (and some ways it's being done to me this semester):

Open book, harder problems

The idea behind this one is to tackle the problem you pointed out directly - how to make a test a test while more-or-less being forced into an open book scenario. In this option, you would simply increase the difficulty of your test appreciably. Perhaps have some questions that, when answered correctly, would confer a C (or equivalent) grade. For the other questions make them incrementally harder. The idea here is to ensure the student is learning something. One way to do that is to push them modestly outside their comfort zone with questions that can't be simply looked up in their text. My program is Computer Science. One way this could be done in my field is to, instead of requiring a student to demonstrate the steps of an algorithm, prove something small. Maybe an example of proving an invariant of something, or a challenging application to something they learned in class. Enforcing a time constraint would be very difficult for someone unless you had a proctor for each individual.

Open book, research

Ask the students to interpret something they've learned in class in perhaps a new light. Depending on the course level you are teaching this could be something very basic all the way to something a PhD student might be expected to do in their research (obviously it must be attainable and reasonable - but I think you get the idea). Again, unfortunately enforcing a time constraint without a proctor would be difficult.

Both of them require more work from you unfortunately. At my level I have not seen any timed online tests mostly because the infrastructure just isn't available to be spun up quickly for classes that weren't already doing it. Additionally, the material is quite complicated and doesn't lend itself well to chunking out into online questions that a computer could grade.

Good luck!

Answer 4

I’m considering making my final exams oral via Skype/Zoom/WebEx/Teams.

My reasoning is that, for the size of class I teach (20 to 24) and the usual exam duration and marking time, it will take me about the same amount of time.

I believe that I can usually ascertain a student’s grade to A/B/C/D/F within about five minutes of questioning. The finer grades will take a little more time, but again still within the time I usually spend on grading and proctoring.

I have a midterm coming up, and will try it out there.

Advantages: you’re pretty sure it’s the right student. They don’t have time to look much up. And any in-room help will be obvious.

Disadvantages: scheduling it may be difficult. Students are not used to it. Getting them to do computations / calculations may be problematic.

—-

Edit:

So far, I’ve done about about one hundred oral exams and it’s worked very well. The scheduling issue has a tech solution: I’ve used Microsoft Bookings which has direct access to my calendar and let’s the students manage their booking. Plus, I get to schedule when they can book their exams.

Answer 5

I would resist the COMMON ADVICE to go for some harder, research-y, project-y type questions. They are not as pedagogically helpful for a neophyte. (Less progressive, less basic skills testing.) Instead I would decide that some loss in cheating is worth the cost, versus a cost in learning.

All that said, the following actions can be taken to mitigate* the cheating danger:

1. Make the exam timed, AT a specific time (this is a good idea in general to exert some discipline...CLAIM your regularly scheduled time), with typed answers submitted in last few minutes of the test. NOTE: you don't need some fancy system. It can be as simple as an email which they "reply" to by adding answers.

2. Make the tests LESS high stakes by doing more frequent AND shorter exams. A weekly Friday hour long exam is not unreasonable for a 5 day per week class. An every two weeks hour long exam is reasonable for a 3 day per week class. Several studies have shown that lower stakes, more frequent tests are less likely to be cheated on.

3. Add a signature block (can be typed name) that no assistance was received. The intention is not to stop ALL cheating. But some fraction of it.

4. Make the exam 10% shorter based on the time for the kids to access the email and reply to it. (And let them know this.)

5. Make it a little easier than normal. The benefit in driving some feeling of success and lessening the "need to cheat" feeling is worth the small loss in challenging the best students. But if you want, you can do an extra credit (with the clear communication to the class, that it will be significantly more difficult and not to attempt it until all other problems have been attempted. Make the stakes relatively LOW (points awarded) for the EC, so it really is just for the kids that are "acing" the easied-up material.

6. Make all the questions very clear and more mechanical. Avoid problem formulations that require explanation (possible, but more difficult/distracting to communicate a clarification with students not in the room). No tricky words. Use the symbols they are used to, etc. In mathematics or chemistry or physics, stick to problems that mimic the examples.

7. For chem/math/physics, it is unreasonable to ask for writing in LATEX or the like. So I would eschew the normal problem style where you check the work and instead ask for actual answers (grade only those, only require the typing of those). But you can compensate by giving more (but more simple) problems or by asking for answers to intermediate steps. For truly essay style questions (English, history, biology), this is less a concern. Just let the kids type (they all type nowadays...but typing an equilibrium problem in chem is not trivial under exam conditions).

8. Probably some things like diagram drawing (geology, biology, etc.) will be more difficult and I would eschew these questions during the "situation". But if you want, you can do the work to devise questions (e.g. give the word that corresponds to A, B, C, D on a diagram.)

9. (For 7 and 8, the rationale is not to expect them to have a scanner or to challenge yourself with grading their smartphone snapped pictures. That will be a nightmare.)

---

Bottom line: 3/4 of a loaf WELL EXECUTED is better than a whole loaf that falls in the water.

*"Mitigate" means partially eliminate; NOT totally eliminate. SE Q&A pedants, please note.

Answer 6

A minor suggestion if you're worried about cheating and this is about objectively graded subjects (as opposed to writing a 2-page essay on Machiavellian philosophy).

With minor programming-fu, send each person a test with a random subset of problems from a collection, and every time a problem goes out, change the value and name of some of the variables. You might jest, but my Dad, who taught college math, handed out 2 different paper exams. A kid got a zero on a problem in versionA, and went to my Dad and said "why a zero? the guy next to me had the same answer and it was marked correct."

Answer 7

I face the same issue.

One point I am considering is time , if you write the questions such that if they answer based on memory then you get a (relatively) structured answer.

But if they spend most of the time shuffling through books then the answer will tend to be “chunks” and “lumps” that don’t perhaps flow as well.

So take care with the amount of time...