2

What research exists (point me toward references) that compares the effectiveness, validity, and security (or at least a subset of these) of paper and online student evaluations of instructors?

At my institution, a working group of faculty and administrators have proposed switching from a paper-based student evaluation system to an online system. I do not doubt that an online system will save time and money. The current system involves printing, sorting, distributing, collecting, sorting again, reading, and processing three pieces of paper for each student enrolled in each course, followed by data entry and number crunching. I lose an entire period of class time when these happen so that we can guarantee most students participate. According to the office that coordinates this effort, the time and cost involved are equivalent to one full-time employee above and beyond what people are already paid to do.

However, many faculty members have raised concerns, among which are the following:

  • Participation will decrease. Only the motivated (very conscientious and very irritated) will participate, potentially skewing the results.
  • In order to increase participation, instructors will be required to offer "carrots" that amount to grade inflation.
  • Since all students on the roster will be notified by email, even students enrolled in the course, but who have never come to class, will be evaluating instructors. These students who have not participated, do have a right to evaluate instructors, but usually do not because it occurs during class time.
  • Student evaluation of instructor surveys will no longer be anonymous. Students will have to log in with a certified log in to verify that they are who they say they are and to match them to surveys for specific classes. While this data will be "anonymized" prior to processing and distribution back to instructors, it will always exist somewhere with identifying information attached to it.

Members of the working group have responded to these and other concerns with statements beginning "Research shows..." However, as an academic, I do not believe "research" until I have seen it myself. The working group has not yet provided a list of references.

  • 3
    While this data will be "anonymized" prior to processing and distribution back to instructors, it will always exist somewhere with identifying information attached to it. - Why would the survey answers tie back to the student? It is perfectly possible to record that a student "voted" without actually connecting answers back to their individual vote. If you cannot create a truly anonymous system, that will lower participation (amongst the very irritated), skewing results. – earthling Nov 30 '14 at 13:18
  • "Participation will decrease." - You can implement a policy like, students who complete they evaluations have access to final grades several weeks sooner than students who don't. (I've attended a school that implemented this, and also what earthling said) – ff524 Nov 30 '14 at 16:52
  • 1
    @earthling - While I understand that this can be done, not all of my colleagues do. – Ben Norris Nov 30 '14 at 21:16
  • @ff524 - That is one of the "incentives" for participation that has been suggested, but faculty do not like holding grades hostage, which we currently do not do with the existing system. – Ben Norris Nov 30 '14 at 21:16
  • @earthling: Certainly it is possible to store the responses only in anonymous form. The question is, how do you convince students that this is really being done? Also consider that there may arise cases where university officials really want to break anonymity; what if a response contains reports of criminal activity by an instructor, or threats of violence or self-harm? Are people willing to have a system that would make such responses impossible to trace? – Nate Eldredge Apr 26 '15 at 22:19
2

There has been quite a bit of literature on this topic. Because your question is broad (regarding acceptability, security, reliability, and validity), I have provided a brief overview; these articles should provide you with a starting point to locate additional information.

Reliability and Validity

A recent article providing an overview of the validity of different student evaluation methods in general may be of interest; the article does address online administration as one component impacting validity (Spooren, Brockx, & Mortelmans, 2013).

In a longitudinal study of over 63,000 student evaluations over 10 years (comparing paper in-class with electronic out-of-class evaluations), no significant differences in evaluation scores were found due to administration method (Risquez, Vaughan, & Murphy, 2015). However, one study had nearly 200 business management students evaluate the same teaching with both on paper and electronically within two weeks (Gamliel & Davidovitz, 2005). Although the mean scores were similar between the two groups, stability of scores was higher in the paper method. Thus, if the same teacher were to be evaluated multiple times, there might be more variance if the method of administration changed. The authors suggest that paper evaluations may allow students to look at the written horizontal profile of their responses, which they may recall on the second administration. In contrast, electronic systems that display questions differently may remove this effect.

A small study of over 120 medical students found that those who submitted paper evaluations immediately after training gave more favorable reviews than those that submitted electronic feedback at a later point in time (Palmer, 2009). The author posits that it is possible the rapport between the instructor and students immediately after session lead to better reviews, while those providing electronic evaluations did not have that social pressure. However, they state it is also possible that the time required to log-in and take the electronic survey may have been irritating, resulting in more negative evaluations.

Response Rates and Timing

Adams and Umbach (2012), reviewed 135,000 online course evaluations completed by 22,000 students. They found that students were more likely to respond to evaluations for courses within their major, and less likely to respond for ungraded courses or if the student was receiving a D or F grade in the class. They also noted some differences in response rates across majors, and evidence of survey fatigue that may impact response rates. Similarly, a study of graduate students using online course evaluations found that those students who responded earlier had more positive course ratings and expected to get higher grades, while those who submitted information later had lower grade expectations and course satisfaction (Estelami, 2014). The author suggests that universities must be aware that early/late responders differ in several specific ways that may impact the data they provide. In contrast, Pegden and Tucker (2012) found that the timing of completing the evaluation electronically did not impact responses, as evaluations were generally stable regardless of when there were completed.

Another article based on online evaluations at the University of British Columbia provides guidance on how to assess adequate response rates to enhance reliability and validity (Zumrawi, Bates, Schroeder, 2014). Nulty (2008) provides suggestions for improving response rates for teaching surveys, and compares paper/electronic methods; he suggests a mixed method approach may result in the highest response rate.

Acceptability

A study of nearly 1,000 students found fewer students completed the electronic version compared to a paper survey; students reported frustration with the site’s complicated and time consuming log-in process and were concerned about anonymity (Dommeyer, Baum, & Hanna, 2002). However, those authors caution that these burdens may be reduced as technology improves. Indeed, a more recent study of over 1,000 students found that students preferred the electronic evaluation option because they felt it was more anonymous, convenient, and presented less time pressure than completing the survey in class (Kinash, Knight, & Hives, 2011). A study of student perceptions about what is important in an online evaluation system reported a list of student concerns related to anonymity, ease of use, and being made aware of the impact of their responses (Nevo, McCLean, Nevo, 2010); this study provides information on a range of student concerns related to the evaluation system, with suggestions for other universities to follow.

Anecdotally, my graduate school recently transitioned from paper to online evaluations. I prefer the online evaluations because they allow me time to gather my thoughts, and I can provide more structured feedback regarding the class. Log-in is simple and is based on the universal log-in for our university; I take it on faith it is de-identified before being sent the instructor. Given how small some of our graduate courses are, that actually feels more anonymous than turning in a paper survey with my handwriting. We receive multiple email reminders to take the surveys during the last two weeks of class, which indicate the overall percent of students that have already responded (a form of social pressure, I suppose). There are only two things I dislike about the system. First, the response period ends before the final exam period begins, so you cannot include thoughts about the exam in your evaluation (the survey ceases to function). Second, responses are only available to the department chair, tenure committees, and the professor being evaluated. At my undergraduate institution all course evaluations were public, so students could read the evaluations of past classes to determine whether and when to take the course.

  • Adams, M. J., & Umbach, P. D. (2012). Nonresponse and online student evaluations of teaching: Understanding the influence of salience, fatigue, and academic environments. Research in Higher Education, 53(5), 576-591.
  • Dommeyer, C. J., Baum, P., & Hanna, R. W. (2002). College students' attitudes toward methods of collecting teaching evaluations: In-class versus on-line. Journal of Education for Business, 78(1), 11-15.
  • Estelami, H. (2014). The Effects of Survey Timing on Student Evaluation of Teaching Measures Obtained Using Online Surveys. Journal of Marketing Education, DOI: 10.1177/0273475314552324.
  • Gamliel, E., & Davidovitz, L. (2005). Online versus traditional teaching evaluation: Mode can matter. Assessment & Evaluation in Higher Education, 30(6), 581-592.
  • Kinash, S., Knight, D., & Hives, L. (2011). Student perspective on electronic evaluation of teaching. Studies in learning, evaluation, innovation and development, 8(1), 86-97.
  • Nevo, D., McClean, R., & Nevo, S. (2010). Harnessing information technology to improve the process of students' evaluations of teaching: An exploration of students' critical success factors of online evaluations. Journal of Information Systems Education, 21(1), 99-109.
  • Nulty, D. D. (2008). The adequacy of response rates to online and paper surveys: what can be done?. Assessment & Evaluation in Higher Education, 33(3), 301-314.
  • Palmer, R. (2009). Evaluation of teaching--paper versus electronic methods. Medical teacher, 31(12), 1094-1094.
  • Pegden, J. A., & Tucker, B. (2012). Does the timing of evaluations matter? An investigation into online student feedback and whether timing has an impact. Studies in Learning, Evaluation, Innovation & Development, 9(1), 55-65.
  • Risquez, A., Vaughan, E., & Murphy, M. (2015). Online student evaluations of teaching: what are we sacrificing for the affordances of technology?. Assessment & Evaluation in Higher Education, 40(1), 120-134.
  • Spooren, P., Brockx, B., & Mortelmans, D. (2013). On the Validity of Student Evaluation of Teaching The State of the Art. Review of Educational Research, 83(4), 598-642.
  • Zumrawi, A. A., Bates, S. P., & Schroeder, M. (2014). What response rates are needed to make reliable inferences from student evaluations of teaching?. Educational Research and Evaluation, 20(7-8), 557-563.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.