I am a PhD student studying wireless networks/telecommunications, and I have developed a set of lab exercises related to the field. We have used them with > 100 students so far in B.S. and M.S. classes on computer networks, wireless networks, and wireless communications.
With the first few cohorts, I asked them to rate each exercise with respect to:
- overall rating
- difficulty of lab exercise
- how interesting the material was
- experiment design
and to rate self on knowledge of topic before and after the lab. I also ask for general open-ended feedback ("How can we improve this exercise?" "Any other comments?") and recently introduced automated systems to measure exactly what students are doing in the lab. This gave me a general sense of which exercises were interesting to students, and which they thought were useful. Now I am looking for a way to evaluate how effective the labs are at educating students - not just their popularity.
I've been doing some reading in CS education research (although education research is not my field, so I can't devote too much time to getting up to speed on all the methodology), but I haven't come across a study design that I think would be a good fit for my scenario. I am under some constraints:
- I'm not the instructor for the class.
- I just run the lab exercises via a website. I don't meet the students in person.
- I don't have the ability to set up a "control group" by offering one section of the course with the lab and one section without.
- I can't ask my students to do a lot of work that is only for purposes of methodology assessment (e.g. I can have them fill out a short pre-class survey, but not much more then that). I can ask them to do some things that are also learning assessments (like quizzes on the lab topic).
Given these constraints, how can I effectively evaluate the educational tools I've developed? (specifically, learn whether they actually improve students' understanding of the course material)
I don't have access to a comparable cohort from one year to the next, or between two sections. In the end, I came up with a set of questions that I think will help evaluate the exercises, even across only one group of students, all of whom are participating in the lab; I describe this in my own answer.