7

I'm a tenure track professor in computer science at a teaching-oriented university. I still have a number of years before I go up for tenure, but want to start preparing early. One of the metrics I will be judged on is teaching effectiveness; namely, some objective, quantitative measure of student learning. I've talked to my (recently tenured) colleagues about this, and it sounds like no one has any idea of best ways to measure it; everyone kind of offered some discussion of test scores, called it a day, and got tenure. I would like to be a bit more principled in my approach, but am unsure how. A pretest seems like the most obvious, since it seems that you would need a baseline to measure effectiveness over the course of the semester. But, I primarily teach intro programming, where students have no background knowledge (those who do are in another class), so giving a pretest seems pointless.

In particular, I have honed in on wanting to teach critical thinking skills (somewhat in line with "How to Think Like a Programmer") and have slowly changed my approach in that direction, but am even more mystified about how to measure that.

In general, looking for recommendations on how to go about assessing teaching effectiveness in a quantitative way in the computer science field.

  • Is this some new policy for which they have not established a procedure? Did you talk to the department chair? – Kimball Sep 7 '17 at 12:02
  • 1
    Yes, I have. Our department (which I love and have a good relationship with) is cynical about all the bureaucratic red tape, including promotion and tenure. My departmental reviews have gone very well, but I'd like to do better for the tenure committee. We've had faculty sit on those committees, and I've talked to them, but they don't have a high opinion for the criteria some other members of those committees use (pre/post tests seem popular in other depts). So, I would like to do something that will not only be convincing for the committee, but also something that I'm proud of/find useful. – cryptic_star Sep 7 '17 at 12:09
  • 1
    Do you mean principled in your approach, or systematic? – aparente001 Sep 7 '17 at 12:19
  • 2
    Related question that I asked some time back (still haven't received any good answers on that one, unfortunately): Alternatives to relying solely on student evaluations of teaching to help administrators more accurately assess teaching effectiveness – Mad Jack Sep 7 '17 at 14:36
  • 1
    @MadJack I think the main issue is that there is no good objective solution to your question (though I think the asking the question is worthwhile). There are various goals in teaching, some of which are long term and some of which are impossible to measure objectively, particularly if you want to apply it to all courses. – Kimball Sep 7 '17 at 16:26
2

objective, quantitative measure of student learning

This is impossible today and probably always will be. "Learning" is too broad for a single metric. Form a specific hypothesis, then test it using a pre-test and post-test. Ideally, you should use two groups of students, one of which receives an intervention. The other group is a control.

You can get free training courses here: https://www.cirtl.net/p/cirtl-programming#Courses (see "Teaching-as-Research")

Do not forget to include qualitative information and feedback from peer faculty.

  • In addition, you can do this with several hypotheses and just report the results that turn out positive for you. I.e., a good metric for you is one that makes you look good. – Kimball Sep 10 '17 at 14:34
2

Consider using a "program assessment" approach to your self-study. Program assessment is normally meant to investigate what students know overall after a complete course of study (degree program). But I think the approach can give some useful information about individual courses, without the wierdness of pre-testing students on a subject they haven't been taught yet.

Basically: Draft a few concrete goals for the course that you teach, and then turn those into specific test questions that you always carry on your final exams. Ideally, you'd draft those goals and questions with input from other faculty members. Record statistics from those final questions each semester. Be on the lookout for any questions where the students are weaker/struggling more than others, and use that as an opportunity to modify or re-focus your course content. If scores go up in semesters after that, then you can hold that up to the tenure committee as an example of your commitment, awareness, and self-improvement in teaching methods. The fact that you have some years before tenure means that you have time for this long-term approach.

Information on program assessment in general can be found here (UMass, "Program-Based Review and Assessment", 2001):

http://www.umass.edu/oapa/oapa/publications/online_handbooks/program_based.pdf

Information on assessment for Computer Science in particular can be found here (U. Pittsburgh, 2014):

http://www.dbserc.pitt.edu/Assessment/Assessments-Computer-Science

  • 1
    I think this is not a bad idea (+1), but just want to point out there are some obvious drawbacks. To give meaningful results, it assumes: (1) you teach the same course with the same focus many times, (2) the quality of students/student support is roughly the same, (3) students don't find out you're using the same questions year after year. In my particular situation, I don't think any of those hold at present. – Kimball Sep 10 '17 at 14:33
0

You still have some years and you teach your students not only what is needed to get through your exam but also critical thinking and skills that will get useful later on. Taking both together, you might be able to get nice statistical results if you keep an eye on your students. Can you maybe find out that the average GPA of students who visited your course went up after that? Can you get access to the number/percentage of your students that got a degree after passing your course (maybe with honors) and compare it to the average?

Of course there might be no significant difference here, but you should still keep an eye on it, as teaching thinking skills often only pays off later; but is still just as important.

Note that this should not be your only argument, as it might not be a good one. But if you can get the standard test scores and also add something here, showing how your special teaching approach was beneficial to the students in the long run, I would go for it.

0

Teaching effectiveness is often boiled down to some measurement of observable change in the behavior of students. As such, you should be seeking to find some behavior that you can observe in your students that have changed in part due to your instructional practices.

I believe that a pre-test post-test approach can work in your intro to programming classes if you compare different teaching styles/methodologies either used in the same class (within group) or different sections of the same course (between group). You first search for different ways to teach your content, run the experiment, and you will then have clear-cut evidence of the best way to teach your own students in the context that you work in. This would be an example of action research.

When it is time to discuss teaching at your tenure interview you discuss your findings and share how you know what is effective teaching for the classes you teach.

In relation to critical thinking, this is really messy to operationalized and measure quantitatively. How to measure a students ability to compare/contrast, synthesize/analyze, and or evaluate is exceedingly difficult as these are actions that take place within the mind intuitively and cannot be observed. Often, we know critical thinking when we see it but can't measure it with numbers because critical thinking is about developing an opinion which is difficult to measure objectively. Documenting this qualitatively is reasonable but may not be acceptable since your department is requiring numerical evidence.

As such, teaching critical thinking skills but perhaps leaving it out of the discussion for tenure would benefit students without complicating your employment

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.