In the (not so distant) past I have received the student evaluations for my course. The evaluation was mediocre, something that surprised me since I thought I had good connection with the students coming to the class (about a third of the total number of enrolled students). These evaluations raised some eyebrows among the relevant committee members.
By accident, chatting recently with a student that really liked and performed very well on the course, I realized that the student did not receive invitation to evaluate the course. I thought it might be an isolated "bug" of the system but I casually mentioned it to a member of the relevant committee. He dug out the system and actually sent me a list of all students that had been invited.
To my shock, 38% of the enrolled students were not invited! What is even more weird is that this missing part is highly non-random and included many of the best students (that got 7+) and only one of the students that got 9+ was invited. The "informal" explanation was that because of some mistake (I do not know if it was the system's or some person's) the students who follow a research track (naturally, the good students) were left out and nobody was actually aware of that. They didn't seem to care even after I pointed it out.
To make it more concrete, out of the 75 or so students, the evaluations included the answer of 26 (non-randomly selected) students.
How can I deal (internally) with this situation appropriately? I have in my record a very mediocre evaluation but it turns out the system made a big mistake leaving out many (good) students. How can I explain this to possible applications where they ask about teaching statement and evaluations?
The main problem is that my department takes evaluations extremely seriously, and my mediocre evaluations had some (admittedly mild, but still...) impact on my development and assessment in the past and might have in the future as well (to some extent, my tenure-track offer depends on that).
Can the department use these evaluations to decide if they will offer me assistant professorship based on the situation explained above? If they indeed use them, is there anything I can do?
Added: The protocol is to invite every student in the evaluation. Moreover, the evaluations should be collected before the grades of the exam are communicated to the students. In the case described above, besides the fact that a particular student group did not receive invitations, these were sent after the students have received their grades. I do not claim that this was done on purpose. But, besides being extremely unprofessional, it seems that no one cares about these facts and they still want to use the mentioned, obviously bugged in my opinion, evaluations.