I'm a tenure track professor in computer science at a teaching-oriented university. I still have a number of years before I go up for tenure, but want to start preparing early. One of the metrics I will be judged on is teaching effectiveness; namely, some objective, quantitative measure of student learning. I've talked to my (recently tenured) colleagues about this, and it sounds like no one has any idea of best ways to measure it; everyone kind of offered some discussion of test scores, called it a day, and got tenure. I would like to be a bit more principled in my approach, but am unsure how. A pretest seems like the most obvious, since it seems that you would need a baseline to measure effectiveness over the course of the semester. But, I primarily teach intro programming, where students have no background knowledge (those who do are in another class), so giving a pretest seems pointless.
In particular, I have honed in on wanting to teach critical thinking skills (somewhat in line with "How to Think Like a Programmer") and have slowly changed my approach in that direction, but am even more mystified about how to measure that.
In general, looking for recommendations on how to go about assessing teaching effectiveness in a quantitative way in the computer science field.