38

How common is it that exam questions, after being designed by the instructor and before being given to the students, are checked by a colleague in one's same institute, to make sure that they are appropriate and unambiguous and that their results are correct? Is this normally enforced by universities across the world?

I am asking whether there is a formal check required by the department; I am not speaking simply about instructors of the same course writing the exam collaboratively.

Motivation: anecdotal. A recent answer mentions "having [exam questions] checked / verified for quality, consistency etc". And I heard from a colleague from a country in the English-speaking world that in their institution something similar goes on: they prepare exam questions one week in advance, and then they are sent to be checked by a colleague. OTOH, this is completely unheard of for me in Italy --- the instructor designs the questions, then they are sent out to the students without any form of review imposed by the university (though the Italian system is somehow peculiar when it comes to exams.)

Improve this question
5
  • 1
    Uh? How do you prevent your exam content from being checked by colleagues, given (at least nominally) the exams in Italy are administered by a committee? – Nemo Oct 29 '18 at 13:48
  • 1
    @Nemo Typically the exam committee is composed of 2 (sometimes 3) people, and they are usually directly involved in teaching the course and preparing the exam questions; sometimes one of them is a TA or a student. I do not consider discussion between the instructors as a real review by a third party. – Federico Poloni Oct 29 '18 at 13:52
  • 2
    Ok. In my maths course, it was not uncommon for the committee to have 2-3 full/associate professors, e.g. the professor of geometry II in the exam for geometry I. – Nemo Oct 29 '18 at 15:10
  • 1
    External review of exam tasks before giving them to the students seems to be common practice in Norway. Also common is external evaluation of the solutions of all students or selected few. – Richard Hardy Oct 31 '18 at 11:30
  • This recent question is an example of why such a review would be a good idea. – kasperd Oct 31 '18 at 23:29

11 Answers 11

Active Oldest Votes
48

Yes; as mentioned in the UK it is expected that assignments (usually those greater than a certain proportion of the overall mark) is both internally checked and externally checked before being sat by students.

I have acted as an external examiner involved in the checking of the exam papers of another university. It is a valuable role that experienced academics can play in both improving the student experience, and the quality of the learning and teaching of an institution.

Another aspect that is also worth mentioning is post-exam error checking. In my department (some considerable time ago) we did an experiment of random error checking and to our astonishment, discovered that despite the diligence of the most experienced marker, things got missed. Our checking showed up whole missed script pages, ridiculous arithmetic errors in the final score and whole class transcription errors where spreadsheet cells were "off-by-one". As a result more regular clerical checks were done on everyone's marking and result recording to gain much in quality and student satisfaction.

Improve this answer
11
  • 2
    Well that’s the UK covered, what about other countries? – Solar Mike Oct 29 '18 at 11:32
  • 8
    Same here in Australia. – masher Oct 29 '18 at 12:52
  • 6
    It doesn't always find errors, though. In my final exams at a "top-5" UK university, two questions on the same paper, ostensibly relating to two different courses, were for all practical purposes identical. There was some discussion among the candidates afterwards about whether the answer "This has already been proved in the answer to question 5" would have got full marks as the answer for question 10! – alephzero Oct 29 '18 at 13:13
  • 2
    Not quite common in France but sometimes yes. I have seen some of my exam questions checked by the professor of the course. – optimal control Oct 29 '18 at 15:03
  • 1
    @alephzero When I was a student, a friend had two questions that answered each other! – Jessica B Oct 31 '18 at 16:53
32

This seems to be very dependent on location. I think it is a good idea, but in the US, at least, it seems to be very uncommon. Lecturers might need to have exams looked at by a more senior faculty member in some institutions, but mostly the faculty is trusted to get it right.

In some institutions that teach very large classes, exams may be created by a "committee" consisting of the main faculty member and a few senior assistants. This gives some assurance against gross errors. In such places you can also have some of the TAs take the exam under exam-conditions. This gives both an indication of overall difficulty and helps guard against ambiguous or misleading phrasing.

Much less common, of course, is to have exam questions statistically validated prior to use. It is very difficult to do such a thing of course unless the exam questions are national in scope and can be trial tested under experimental conditions.

Improve this answer
8
  • 1
    So, there are never any questions that cannot be solved, or questions that have information missing or questions that are ambiguous... All confusing for students... – Solar Mike Oct 29 '18 at 12:14
  • 11
    In my experience it's also fairly common to have the TA(s) look over the exam, if the course has any. Whether that counts as peer review is of course up for debate. – Anyon Oct 29 '18 at 12:15
  • 4
    Some TA's are very good at finding the typos (errors) before the students get to sit the exam... – Solar Mike Oct 29 '18 at 12:16
  • 2
    @SolarMike, you've read enough questions here, of course, that you know that bad things do happen. In some smaller departments there might not be any other faculty member with the expertise to actually answer some of the questions. They can help with preventing double-negative phrasing and such, of course. – Buffy Oct 29 '18 at 12:18
  • 4
    One reason for the difference is that a final exam might typically count for 25% of the grade in the US and 100% in the UK. – Noah Snyder Oct 29 '18 at 12:28
13

Very common in my experience. just to note I am in Switzerland...

And even if it is not prescribed by the university, common sense does seem to get most of my colleagues in the past to ask one to "have a look at x" on an ad hoc basis. We tended to do this for each other, even before "moderation systems" were "forced upon us" or the existing relationship was made more formal...

This, simply, helps to avoid typing errors, or number errors ie a factor of 10 missing that can make a really good question such an issue for students who are under stress...

Part of moderation can be where external moderators from other institutions come in to evaluate a particular course, where they speak to the lecturers, the students and check the exams and answer scripts for consistency.

On a personal note, I did not like being externally moderated at first, but now value the process for a different view point as I have found the people who come in face the same problems with classes / students that I do. So, the discussion tends to be "I'm considering this" and the response is " Well, interesting, worth a try, but look out for this or this". It makes a difference which I now value...

I remember a Professor, where I did my studies, who was told to write an "open book" exam for his course. So he duly did so... Sent it in for external moderation (the so-called experts as mentioned in another answer...) and the moderator could not complete the exam... :) The moderator had to ask for the solutions... My lecturer continued with the original style of exam not going to open book....

Improve this answer
0
8

Definitely location and resource dependent. I have never seen this done for midterm assignments or exams at the two institutions where I've taught (USA community colleges in Boston and New York; 5 years, 12 years respectively). For final exams my current department does have uniform department finals, which are viewed by all instructors beforehand.

Arguably institutions like education in the USA tend to be relatively "throw everyone individually to the wolves/law of the jungle", and this might be one example of that.

So, it definitely seems like a nice idea that I've never seen implemented. (Exactly like code reviews, back when I was a software engineer.)

Improve this answer
1
  • 1
    Same in Canada. I had one professor who reviewed questions that a surprising proportion of the class got wrong, and gave partial credit or discarded the question, but that was about it. – Matthew Read Oct 31 '18 at 19:46
8

In the United Kingdom, the quality assurance process imposed by the UK Quality Assurance Agency requires both internal and external moderation of assessments. Internal moderators are academics from the same unit or department as the lecturer. External moderators are academics from outside the university (and may be international experts in the field).

Internal and external moderation occurs at two stages. The first is in the construction of exams and coursework assessments prior to release to students. These assessments are subject to review and revision before release. The second is in the assessment of the distribution of marks and in the conduct of the exam. In this step, we discuss issues such as the performance of the cohort against expectation, cases of academic misconduct, etc.

The regime is quite involved and very bureaucratic. However, its value is apparent when there are disputes about the level of the exam or the performance of students.

Improve this answer
6

In my experience, this is a question of the amount of resources a course is allocated. If a course is budgeted properly, the teaching staff spends time on properly taking care of the website; preparing and updating course booklets or interactive on-line teaching materials; reviewing each others' slide decks; reviewing draft homework assignments before publication; reviewing exam questions and so on.

In fact, when I was a TA for first-semester programming course (234114 and 234117 at the Technion IIT), it would often be the case that TAs who did not participate in writing exam questions would sit the exam a while before the official exam date, to ensure that the instructions are clear and also to time themselves. We were aiming for a 3:1 ratio between the time the students are given and the time a TA writes up a complete solution.

So, in particular:

  1. There is no dichotomy of "university review" vs "no review".
  2. Sometimes the TAs write exam questions, not just the teacher in charge.
Improve this answer
2
  • Hmm. You were stricter than us in the US. We usually aimed at 4:1 or more. Though that was for Calc I, Calc II where I think the amount of menial work compared to thinking is lower. I.e. you need to write less for a complete solution if you know what you should do, so the TA's had a bigger advantage. – DRF Oct 31 '18 at 16:17
  • @DRF: For Calculus, there's a lot less writing than for a C programming course where it takes a while to write down full programs. – einpoklum Oct 31 '18 at 16:29
4

In the Czech Republic (Charles University at least but most reputable ones from what I know) we still have most (90%+) courses graded as part of oral exams thus avoiding this kind of problem neatly, while also actually figuring out what the student knows as opposed to whether or not he studied for the test.;) Some of the exams have a written part that is succeeded by an oral examination though.

For the 10% that are purely written I've never seen external validation though internal validation between colleagues happened occasionally.

Improve this answer
4

I work for an Australian university and yes, we need to submit the final exam to the department for approval two weeks in advance. The purpose is to make sure that the questions are unambiguous, comprise a range of the cognitive levels and use the right format and paper organisation.

Not only the exam, but even the marks need to be moderated by other academics in the department.

Improve this answer
1
  • @Federico Poloni: apolgies I meant to say two weeks. Two weeks before the the examination period starts, though, not the actual exam. – ThunderDownUnder Oct 30 '18 at 12:02
3

Where I work as a PhD student in Germany, the exam questions are designed by me and then checked by my supervisor who is giving the lecture. So, there is no "outside check" system but the exam gets checked by different people. However, different professors handle that differently (for example, some of them design the exams themselves and then their PhD students check them or a colleague checks them. It is, however not even unheard of that someone designs an exam and doesn't let anyone check it. That this is possible is quite unfortunate in my opinion.)

Improve this answer
2

While I was a PhD student at a German university, this was absolutely common. My exercises - and those of my colleagues - were multiple times checked by colleagues and rarely adopted in their original form.

It was fairly normal that the exam questions had to be revised because the exercise was simply wrong. Often enough, the formulations were unnecessarily complicated or ambiguous - or the exercises required mathematical techniques, which the students didn't necessarily possess at this moment.

Sometimes the exercises and corrections were challenged in court by students who saw this as a last resort to stay at university.

Improve this answer
2

I do think it is common and want to add another aspect: Time.

Often one or two colleagues are given the full exam and they answer it, noting the time needed for every question. Afterwards the time estimate can be multiplied by some reasonable factor (e.g. between 1.5 and 2), to estimate what you can expect from the students during the exam. This can filter too long exams and help to weight the points for each question.

Improve this answer

Not the answer you're looking for? Browse other questions tagged or ask your own question.