4

I am currently the course leader of an introductory course for undergraduates. Each semester, more than 500 students take the course. One of the course assessment tasks is a multiple-choice quiz which is held twice a semester.

One of the problems is that the quiz is currently paper-based, which makes the quiz logistically challenging. For example, we have to reserve multiple large lecture rooms, we have to print question papers, we have to collect multiple-choice answer sheets and scan them. In addition, it is not environmentally friendly to print and destroy such a large number of pieces of papers (our quiz consumes a few thousand sheets of paper!).

I am thinking of moving to a computer-based multiple-choice quiz, which will be held in a computer lab in the university. Unfortunately, the maximum capacity of a computer lab is about 80 people, and due to the limited number of computer labs available, we would not be able to give the quiz to all of the students at the same time. In order to avoid the students who take the quiz earlier leaking the questions to the students who take the quiz later, I would have to design a test bank, so that the computer system would randomly draw questions from the test bank to assign them to students.

Questions:

  1. Would such a system be "fair", given that each student is taking a "different" test?
  2. Do I need to design a very large test bank to protect against students who take the test early sharing the questions with students who take the test late?

I would especially appreciate it if teachers who had successfully administered computer-based quizzes could share their best practices.

  • You might want to take a look at Google Apps for Education. – aparente001 Nov 22 '16 at 23:12
  • @aparente001 Could you please tell me more about why I might be interested in Google Apps for Education? – I Like to Code Nov 23 '16 at 10:15
  • I haven't experienced it from the instructor side, but I've seen it from the student side (K-12). Google has built a massive educational platform, and online quizzes are one of the features. // Have you thought about foldable cardboard privacy carrels, by the way? // Also, softwarerecommendations SE might be a site to work with on this. (You're not allowed to repost your question verbatim, though.) – aparente001 Nov 23 '16 at 13:39
3

During my undergrad we had similar testing for a lab course. The faculty had a database of questions grouped by concepts and scored with the relative difficulty. A program would pick a given number of them from each concept (with regard to the group's lab content, specified by faculty) with the constriction that the total difficulty score was set to some given value (usually the max points of the test).

As for the initial seed of the database, they simply took a few years back of paper-based test questions, when they first introduced the computerized system. Although, it was substantially expanded with time.

By the time I took the course, they were confident enough with their database, that each student got their own questions during the test. Also, note that these tests were done by each group before each lab appointment, so that is a lot of students spread over a week or so.

The bottom line is that you don't need an extremely large database to begin with, you just need to keep expanding. You also can start simple and increase the complexity of the system (e.g. by introducing random values to the questions, as already suggested).

The system is fair from the point that the students have to know at least for a passing grade for questions from each of the course's topics to be confident to pass. It is unlikely for a student to read, say, 2 out of 5 chapters and get lucky enough to get questions covering exactly the subjects he learned. Although he can be reasonably sure that the topics will come up, he can't be sure of the difficulty. It is usually easier to shallow-read the whole required material and be prepared for the easiest questions from all the topics, than to in-depth study one or two of them to be confident to get the hardest questions right. Both kinds are bound to come up on the test.

3

What you describe is pretty much exactly what we do with final exams in our large-section math courses at my institution. For a quantitative-heavy subject like ours, the following is feasible: effectively ask everyone the same questions but with different numerical values involved.

Our elementary algebra course uses the Maple T.A. platform, where question formats can be coded and somewhat different numerical values presented to each student.

Our remedial arithmetic course uses the Blackboard testing system, which does not support randomizing values for fraction or radical-form answers (but could for decimal-form answers: see questions of type Calculated Formula), so the faculty have manually drafted about a half-dozen separate, parallel tests of equivalent difficulty.

The Maple product is definitely a better system for this kind of setup, but it may cost money, whereas Blackboard is potentially already on site. In each case, the order of questions, and the order of responses, can be randomized for each student.

The overall process works tolerably well. Downsides: To my mind, the questions are rather obviously functionally "the same" for every student -- but the students (at least at this level) don't seem to perceive them this way. Also, because the overall scope of the questions is fixed, in some sections it leads to rather degenerate teaching-to-the-test, and students in the next course in the sequence don't really have any understanding or depth with the skills in question. (For example: Some instructors only teach checking multiple-choice solutions to equations, not actually solving them, etc.) In any case, quite a bit of work is involved setting up your tests.

  • When do the students take the final exams? Do you for example book a computer lab for two whole days, which each student scheduled in a say two-hour slot? – I Like to Code Nov 22 '16 at 13:19
  • @ILiketoCode: Yes, that's exactly what happens, which is always a scheduling challenge around finals week. E.g., this semester one of our courses has to do that that process a few days before the semester is officially over. – Daniel R. Collins Nov 22 '16 at 14:44
  • We also have a large dedicated testing center, but they have other responsibilities as well (entry placement tests), and only run the Maple T.A. tests; Blackboard tests are in our computer labs. – Daniel R. Collins Nov 22 '16 at 14:46
  • "Some instructors only teach checking multiple-choice solutions to equations, not actually solving them" Could you offer ranges as the possible answers, rather than specific values? – Patricia Shanahan Feb 11 '17 at 23:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.