Academia Stack Exchange is a question and answer site for academics and those enrolled in higher education. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I teach physics at a community college and have been on a number of hiring committees for tenure-track jobs. Usually we interview about 6 people for such a position, and of those, usually something like 3 of them show a lack of knowledge of the subject in the interview. When I say a lack of knowledge, I mean that they can't do easy freshman stuff. We will ask them a question that is posed as a request for them to pretend we're a class of students and give a short explanation of something, so it's nominally a test of their ability to teach it, but in reality we find candidates who simply don't know it. These are people who have a PhD from an accredited school. (We hardly ever interview people who only have a master's.)

[EDIT] (To clarify, I'm talking about very basic knowledge, such as understanding what Newton's laws mean. We're not asking candidates to recall obscure trivia. Personally, I don't really care whether they know which is Newton's 1st law, 2nd, or 3rd. I'm talking about candidates who actually demonstrate elementary misunderstandings of Newton's laws. Since a lot of academia.SE users are in math, I think a good analogy would be if someone applied to teach math, and that person didn't know the chain rule -- which I have heard from colleagues in math departments at community colleges is also common. Continuing this analogy, the issue is not that they fail to remember the term "chain rule." The issue is that if they're asked to differentiate sin cos x, they can't do it. They do silly things like attempting to use the product rule, as if the sine was multiplied by the cosine. Or they throw up their hands and won't try, even if the committee tries to help them out.)

Are there good strategies for screening these people out at an earlier stage, without needlessly losing too many good candidates from our pool? Here's what we're already doing:

  • We require applicants to submit undergraduate transcripts, and we are less likely to interview people who have poor undergraduate grades (such as Ds and Fs in math and physics).

  • We prefer applicants who have graduate degrees from more prestigious schools.

  • We prefer a candidate who has taught a wide variety of courses to one who has only, e.g., taught mechanics.

Undergraduate grades do seem to correlate with what we see in the interview, but it's also possible that someone started out their undergraduate education with a weak high school background and then overcame that disadvantage. There is also the difficulty of comparing different countries' grading standards. We require graduate transcripts, but I find those hard to extract useful information from.

We are currently only asking them to give the names of references, but not to supply letters of reference along with their applications. Would it help to make them send letters?

You would think that someone with really weak competence would never get into a good graduate program, and therefore we could just not interview people who have degrees from low-prestige programs. However, our pool doesn't usually include a ton of people who have degrees from the best graduate programs, and we have also seen people in the past with degrees from renowned universities who nevertheless displayed major gaps in their knowledge, as well as highly competent people who got their degrees from no-name schools.

share|improve this question
9  
Yikes. How easy are we talking here? (I for one have forgotten the details of what I learned as an undergrad that I don't use regularly, although I would be able to refresh my memory very quickly if I had to.) – ff524 Mar 1 at 20:45
16  
If you asked me "what is Newton's second law", I'd have to guess. I know what Newton discovered and how to apply it, but not which "law" is which or how they are usually worded. If you asked me to succintly describe the chain rule, I'd also get flustered. I do applied mathematical modelling in a biophysical context for a living, but so rarely use this basic stuff that it isn't at the forefront of my mind. It would take me less than 30s to look them up and have immediate understanding, though. And if I were teaching, obviously this would be kept fresh. – Significance Mar 1 at 22:16
7  
In my experience within my field, I've encountered a lot of researchers from top-tier university that have a fundamental lack of a solid background in the subject matter. In fact, I have found this to be more common from the top tier universities. I think this is because a lot of PhD programs put them directly into research without having them take fundamental core courses. This problem is magnified in my field (statistics) because PhD programs accept students without a statistics background into our program. So many PhDs only seem to know their narrow research topic, and not much more. – Cliff AB Mar 1 at 23:23
5  
@BenCrowell: yes, my guess is that it's field dependent. In biology (my wife's field), it seems the standard is that they take a total of 6 graduate courses for their PhD (real training is in the lab). I was required to take 15 graduate courses for my PhD. But I know of a stats PhD who has only taken 2 statistics courses ever: once in undergrad, once in grad school! – Cliff AB Mar 2 at 0:31
5  
@CliffAB That seems extremely atypical. The vast majority of (good) stats PhD programs require a solid core of classes and have comprehensive qualifying exams before research begins in earnest. Sure, stats programs take plenty of compsci/math/etc. undergrads who haven't taken extensive statistics, but if they don't catch up enough to pass quals then they won't stay in the program. – Roger Fan Mar 2 at 6:15
up vote 31 down vote accepted

Presumably your interview is doing a good job screening out the individuals who you feel "don't know the subject" and you are trying to screen them out prior to the interview. I think a reasonable screening tool could be a a phone interview. You should probably conduct between 10-15 phone interviews to find the 6 candidates you want to interview.

While I say "phone interview", it is most often now a "Skype" interview. These interviews could be as short as 30 minutes and should be no longer than 1 hour scheduled back-to-back such that they all get completed in two long days. The phone interviews I know about have had 3-6 people (presumably the majority of the committee) present, but I think depending on department politics that you could reduce it to a smaller group (maybe even just the committee chair).

While you could simply focus the phone interview on the questions that cause candidates the most trouble during on campus interviews, given 1/2 your candidates have difficulties recalling key concepts of "random" classes, you may want to help them out a little. If you require that applicants submit a sample syllabus for freshman physics and another for an upper level elective, then the phone interview should focus on the teaching statement and sample syllabus.

During the phone interview you should also mention some of the other key classes of you department that they might teach on. If a candidate is truly clueless, this will not matter, but good candidates will realize that they need to brush up on those classes for the on campus interview.

In summary, an inability to talk about a syllabus the candidate has written would be your screening tool and the hints during the phone interview will help you minimize throwing out potentially good candidates.

share|improve this answer
    
Interesting suggestion. Have you done this, and did it work well? – Ben Crowell Mar 1 at 22:43
7  
@BenCrowell for R1s in the sciences teaching is not a priority, but my wife is in the humanities and they require sample syllabi and use phone interviews extensively. They also often have to teach a class during the on campus interview. From what I gather from her, the phone interview focuses on teaching more than research. – StrongBad Mar 1 at 22:48
9  
+1 I think another important aspect in the phone interview is to not let the candidate dictate the entire discussion. I have talked to many PhDs with a very thorough understanding of a very narrow section of their field, but amazingly little breadth. If they spend the whole time talking about the area they know, you may miss out that this is all they know. – Cliff AB Mar 1 at 23:27

You think that asking one or two questions on the spot in an interview is a good way to assess an applicant's knowledge of basic concepts in the discipline. However, this sounds like a relatively poor approach to me.

There are basic principles of psychometrics that are relevant here. Basic principles of item response theory are relevant. First, people differ on a distribution in latent ability (i.e., knowledge of the domain). Second, items differ in difficulty. Third, even with these two bits of information, there is a random component. So if you want to measure latent ability in applicants, then you can improve your assessment by having more items. For example, give them a written test with a bunch of items.

More importantly, I think you need to be be careful with assuming that most people can recall basic facts from the undergraduate curriculum. If you teach that curriculum, then you are likely to recall many such details. If you don't teach such material, even if you've learnt the material and would know how to look it up and solve problems when required, it may not be available to immediate working memory. It might be easy to relearn and if you were given the task of teaching a class related to that content, you may still be able to do a good job presenting it.

In summary, you may want to think about the degree which being able to recall undergraduate knowledge at the time of hire is predictive of job performance. I imagine that it might be relevant, but I also think that a whole range of other indicators would be relevant too. So in addition to trying to measure the construct better, I'd recommend that you treat it as one of several indicators of potential job performance.

share|improve this answer
    
I upvoted for the general theoretical background, but any answer which amounts to "do lots more work and maybe scare more people off" is problematic for the OP, I imagine. To the extent that this means "you will lose some good candidates with your current approach, and here's why" I think it's fine; but beyond that, the "no easy solution" part is IMHO more of a comment than a proper answer. – tripleee Mar 3 at 5:04
    
I guess it could be elaborated. But I think there are a few specific suggestions in the answer: (1) include a written test of knowledge of relevant undergraduate subject matter, (2) weight any ad hoc teaching exercise as just one sign of potential competence within the context of other indicators (undergraduate grades, interview, references, cv, etc.). – Jeromy Anglim Mar 3 at 6:46
1  
@JeromyAnglim I am pretty sure that any department that required me to take a written test would automatically fall to the bottom of my list and would only take a job there if I had no other choice. – StrongBad Mar 3 at 16:04

There's a famous test in the CS world called "FizzBuzz" that companies these days are using to efficiently screen out candidates that managed to get degrees without actually learning anything. FizzBuzz (and its variants) focus on what are considered to be core concepts that any programmer regardless of language should be able to do in a matter of minutes. As reported, the results are astounding. While material on the effectiveness of FizzBuzz in recruiting has never been extensively peer-reviewed (to my knowledge), coming from that world and even being a self-taught programmer I can tell you that anyone worth their salt should be able to complete the test in at least one language, even if the solution is inelegant.

It sounds like you're running into a similar problem in your neck of the woods, and are already onto a similar solution. I'd recommend trying to find that core set of problems that any undergrad should be able to solve one way or another and use that as your screening. Let them use a textbook, maybe hint at a list of possible problems ahead of time as others have mentioned so they aren't caught off-guard, and be willing to help the way you've been doing. But make the first interviews (possibly over Skype as was also suggested) quick and to the point so you don't waste time. Establish either the base knowledge, or at least the ability to pick up on it quickly while under pressure, before proceeding to full, in-person interviews.

Those tricks have worked in my industry, hopefully they can help you in yours. We've seen that it doesn't matter who graduated with what degrees and has what recommendations or grades, the ones who can actually live up to their title and do their job are sometimes the least-decorated.

Edit: These "quick" interviews can be done on Skype in at most 30 minutes, possibly more like 15 depending on the questions. You can knock out a lot of candidates in a short period of time before dedicating serious resources to second-round interviews (which may still be online and only last an hour), assuming you think the process through in the beginning.

share|improve this answer
19  
Well, such tests are not common in academic hiring, and academic job candidates are not used to having their content knowledge questioned at such a low level. If OP's institution adopts this, I would be concerned that they might alienate good candidates, who might, rightly or wrongly, find such tests insulting. – Nate Eldredge Mar 2 at 3:24
7  
@NateEldredge: Good candidates will never be insulted by such tests. Either they will show the interviewers a clear mastery of the low-level content and then it is time to move on to higher-level test questions. It can only be insulting when all the candidates can solve all of the low-level questions and the good candidates are rejected while lousy candidates are accepted. – user21820 Mar 2 at 5:27
13  
The idea is interesting, but I also share Nate Eldredge's concern. I already worry that we're alienating candidates by requiring them to submit an undergraduate transcript. (When I interviewed for my job, I thought it was absurd.) In our interviews, we usually mask at least some of the content-knowledge questions by phrasing them as tasks where the candidate is to pretend to explain the idea to students. This helps to avoid insulting the candidates. A common phenomenon throughout academia is that lower-status schools mimic higher-status ones. – Ben Crowell Mar 2 at 6:27
7  
@BenCrowell I think the point that this is a community college is somewhat relevant. It seems irrelevant to submit an undergrad transcript when your understanding is that you are being hired to do research. If the position is more obviously teaching-focused I would find it much less surprising. – Jessica B Mar 2 at 7:14
3  
@user21820 They may not find them insulting, but they will definitely find them a waste of time. They would definitely avoid the said institution if they can. Imagine this became a standard, and with each application you had to solve a separate low-level task. – Yet Another Geek Mar 2 at 11:29

Learning about a subject is not so much about being able to reproduce that knowledge at any moment, as much as it is about being able to re-learn more quickly the next time around. At some point all it will take is a quick look at the materials, or a little bit of time to re-think the steps involved. It would be a pity to lose out on good candidates because they can't pull up knowledge immediately - and in a stressful situation.

I would suggest giving them a list of possible topics in advance. But a big list, and not much in advance - so that knowledge still plays a major role. Or, alternatively, giving them the option to briefly consult a textbook on the spot. The ability to understand things deeply should still shine through.

I guess this is the opposite of what you asked for - it's not about how to screen people out, but how to screen them 'in', in case they do know the materials and the particular interview situation is not well suited to assess that.

share|improve this answer
19  
This would be a fine answer if I didn't really mean what I said in my question. I meant it. We have candidates showing a lack of understanding of very basic material, such as Newton's laws. We're not asking them to recall obscure trivia. – Ben Crowell Mar 1 at 21:47
3  
@BenCrowellm the FizzBuzz test mentioned above is a trivial application of basic programming skills. Like using Newton's laws to compute e.g. how perfectly elastic bodies will move after a collision. Or perfectly sticky ones, if it comes to that. I'd probably think it is a quaint question, and move on (no, I don't do physics, I'm in computer science). – vonbrand Mar 2 at 12:03

So you have a set of screening techniques, but they are not enough, and you may need to add another one (most probably a phone screen). You have a concern that doing so will alienate the candidates, which is totally understandable.

What you should do is not only introduce a new screening, but remove the ones which don't work. If undergrad transcripts show a correlation with the quality of candidates, keep that requirement. If candidates with PhDs don't show significantly higher quality over people with Masters, dump that requirement.

And don't be afraid that non-standard requirements and tests will be perceived as offensive. What is truly offensive (or, I'd rather say annoying) is wasting too much of someone's time over too small of a probability that you'll actually hire them. Asking to fill an hour-long form before even talking over the phone is offensive, asking to derive sin(cos(x)) is not.

share|improve this answer

To distill some of the other previous comments, I would suggest having a low bar that you expect any viable candidate could do over the phone. The important part is making it easy enough that you don't screen out nervous competent candidates, but it looks like this isn't an issue for your specific situation.

You might frame it in terms of "an initial phone screen"; it also provides an additional opportunity to get to know the candidates.

Also note this doesn't have to be direct questioning, but can be included in other questions (such as "How would you teach Newton's laws").

share|improve this answer

protected by StrongBad Mar 2 at 3:36

Thank you for your interest in this question. Because it has attracted low-quality or spam answers that had to be removed, posting an answer now requires 10 reputation on this site (the association bonus does not count).

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.