10

A recent question mentioned the importance of citation counts and impact factor in evaluations of faculty candidates. There are some good answers explaining why this is the dominant model today.

I'm curious if any other models have achieved a reasonable amount of use today, even if they are not remotely the most common ones. That is, are there models that either do not take into account citation count and impact factor, or where they are relatively minor factors? For example, something like this could match:

Welcome to the Slightly Northwestern South-By-Southwest University of Central Eastern Ruritania. As you wrap up your PhD, note that if you want to continue on as faculty here, your citation count and impact factor are less than 5% of the evaluation. Nobody here worries about them. If you want to get an appointment, you need to spend at least 75% of your time studying for the New Faculty Literacy Exam rather than doing research not strictly required for your degree! Write one trash paper for a third-rate journal if you must, and then start studying! The exam is really tough! Here's an exam syllabus to get started.

To be clear, I'm not asking about alternative pathways to degrees (e.g. degree by exam, degree by prior publication, etc.), and I'm not asking for a list of universities that disregard citation count, etc. in favor of other factors. I'm simply asking if this is actually a thing in 21st century academia.

2
14

Here's what my department will do before offering you a tenure-track or tenured position: Our personnel committee will get informal opinions about you from colleagues in or near your area. You will be invited to give a colloquium talk and possibly also a more specialized talk, and to meet with faculty members here (usually including the chair of the personnel committee and the chair of the department). All of our faculty will be invited to give their opinions after your talk(s). We will, after getting your permission, solicit letters from experts in your field about your research; these will not just be the ones you listed as referees. After these letters are received, the personnel committee will vote on whether to make an offer to you. The committee will also prepare a report about your record and reports on two of your (more important) papers. Then the department's elected executive committee will vote, and finally the whole tenured faculty of the department will vote.

The preceding is what the department will do. If the decision is to make an offer, it still needs to be approved by the college's executive committee and, in the case of a tenured position, by the provost.

5
  • 3
    While the general gist of this is accurate, I think this is more effort than most departments will put into the hire of a tenure-track candidate, though probably accurate for a tenured candidate. Nov 27 '21 at 23:14
  • "vote on whether to make an offer to you" or "vote on which of the N shortlisted candidates to make an offer to"?
    – avid
    Nov 27 '21 at 23:58
  • @avid We vote "yes or no" on each individual candidate whom the personnel and executive committees have proposed. (Often there is only one such candidate, but other names may come up in the discussion.) Nov 28 '21 at 12:59
  • 3
    Blimey! This really is the gold standard, but i've never been involved in any department that goes to that much effort (or has that much democracy). Nov 28 '21 at 13:47
  • I concur. This is similar to how we proceed in the university I work in ( Canada )
    – BlaB
    Nov 29 '21 at 11:41
12

At my institution (a research-intensive Canadian university), the "evidence of scholarship" component of assessment for tenure and promotion is not based closely on numeric summaries of citation count and impact factor. It is more holistic/subjective (which could be argued to be a bad thing, as it is less transparent and allows more scope for bias, but I personally think it's a good thing). For the "publications" component, some of the criteria considered are:

  • volume (number of papers, and consistency of output)
  • authorship (first or last/senior? out of how many authors total? we also value co-authorship with a candidate's trainees, but this falls under "mentorship" rather than "scholarship")
  • quality of venue: we don't usually look directly at impact factors, but rely on the judgment of someone who knows the field (a reference or committee member) about whether the journals are high-quality/high-impact journals for a particular field. For better or worse, "tabloid" publications (Science/Nature/PNAS/etc.) do carry some weight ...
  • judgment of reference letters about the quality and impact of the work, either of individual papers or as a body

We don't request any that candidates include information about impact factor, total citations, or h-indices in their folders, although this information is sometimes provided by candidates or by reference writers — we try to take this information with a grain of salt.

You could say that the criteria listed above are basically the same as "citation counts" and "impact factors", but subjective.

There are also criteria other than publications (presentations at meetings, invited talks).

Similar criteria are used for hiring decisions (especially at the state of preparing a short list of candidates to invite for an interview, before the process described in @AndreasBlass's answer takes place).

See also this answer.

3
  • What do you do when you don't have enough people on the committee that know the field. I doubt anyone on the committee knew anything about my field, Nov 28 '21 at 13:45
  • @IanSudbery: rely more heavily on the letters of reference (which are, by definition, from people who know the field well)
    – Ben Bolker
    Nov 28 '21 at 14:00
  • HR departments hate references as it opens them up to all sorts of issues with accustations of discrimination. Nov 28 '21 at 16:38
9

To echo the other answers: at least in math, in the U.S., appraisal of candidates for faculty positions is much more holistic than just a bunch of numbers. If nothing else, various candidates typically have very different trajectories, making any simple "scalar" comparison ridiculous and meaningless. (Well, sure, unless we just give up and "declare" some number created by a corporate publisher to be what we care about!)

In math, citation indices are a very, very volatile measure. For one thing, after a breakthrough paper that is refered-to for a while, the myriad "secondary" papers will have more relevant technical details... and the original is not cited any more.

Also, nowadays, quite a few people have realized that, for example, a "good mathematician" is more than "just a good theorem-prover". Plus, why hire someone, pay a salary, given them an office, and put up with annoyances, if we can just read the preprints and papers for free? :)

Yes, some people (not only administrators) over-simplify by using grant numbers as a bottom line. This has never made much sense in math, since we don't really need giant machines, etc. Still, some math people like "number of dollars" as a measure, because it is a scalar, and numbers of dollars can be compared... and context ignored, "for simplicity". :)

For that matter, even now, and quite a lot more so 30-40 years ago, the contribution that a math person might make was very subtle to judge. Some things were never "published" at all, but had huge influence. Sometimes people gave unrecorded talks that gave evidence of terrific ideas, and experts had heard the talks, and could strongly recommend the person for jobs. I think "expert opinion" is still the fundamental criterion, not paper-count nor impact...

(Yes, "expert opinion" can also become caricaturized/parodied, into insisting that only far-away big-shots' opinions matter... Nevermind.)

6

The majority of universities in the United States have teaching as their primary mission and hire professors accordingly. While a PhD is required (or de facto required) for most positions, and having a publication or two is an advantage, the people hiring are primarily hiring based on how good an instructor you will be for their students. This is based on some or all of:

  • The teaching experience you have.
  • What you write in the statements with your application about how you teach and why you teach that way.
  • Teaching materials you may be asked to provide with your application.
  • What your letters of recommendation say about your teaching.
  • How you respond to interview questions about teaching.
  • A teaching demonstration, meaning a lesson you give either to a real class or to a pretend class, when you interview.
  • A presentation to a general audience you give during your interview.

Note that students at different institutions are different, and hence someone could be a good or great teacher at one university and merely mediocre at another. This means the evaluation of teachers is localized; the hiring committee will be trying to discern from the information above whether you would be a good teacher at their university.

1
  • 1
    Yes, this is absolutely correct, for the majority of subjects! Certainly for mathematics. For engineering programs it is a little trickier, given the enormous costs of equipment. Nov 27 '21 at 23:19
5

This is a fair question, however it is based on a wrong assumption as I've explained in the original question referred to.

To repeat: NO credible university I am aware of, globally, considers directly impact factors of journals as a measure by which faculty members are being assessed.

All universities asses academic staff by intertwined, multivariate, different metrics:

  • reference letters and peer evaluation
  • places of publications: prestigious or not
  • acquaintance of existing faculty members with the candidate (for recruitments)
  • number of publications in selective places (not number of citations)
  • service and teaching
  • outreach
  • international recognition based on invited talks, editorial membership etc.
8
  • I must admit I don’t quite follow. You claim on the one hand that no organization considers directly impact factors of journals yet isn’t your second item basically just that, except you don’t put a number figure to “prestigious or not”? Nov 29 '21 at 1:38
  • 1
    "NO credible university I am aware of, globally, considers directly impact factors" should read "NO credible university I am aware of, globally, admits to considering directly impact factors" Nov 29 '21 at 3:20
  • 1
    @ZeroTheHero, No. Prestige is a pre-determined set of journals and conferences. It has nothing to do with published impact factors (which are many times fake). Everyone knows that Nature and Annals of Mathematics are prestigious. No one checks their IF. And the same goes further down the line in journal prestige.
    – Dilworth
    Nov 29 '21 at 17:18
  • 1
    @AnonymousPhysicist, no, I actually think that almost no one checks the published impact factors of journals. These are unreliable numbers.
    – Dilworth
    Nov 29 '21 at 17:19
  • 1
    @ZeroTheHero, I'm not against rankings and linear-orders of journals and universities. My claim is different and specific: Impact factors are not used nor regarded as an important ranking. Other measures are important, but not IFs that are ignored in general.
    – Dilworth
    Nov 30 '21 at 15:29
2

Others have covered most of the obvious points.

In addition to those, we try to find someone who we can get along with and who will likely do well by themselves. Thus, beyond being a “good fit”, we look for compatible personalities and evidence of independent work.

We do this through the cover letters, by asking candidates to supply samples of their papers, by reading reference letters, by reading other recent papers of the candidates.

On the research side, is the candidate a cog in a large machine or was he or she significantly involved in the work? We often ask candidates to describe their role in these publications, to clearly understand attribution. On the teaching side, how well does the candidate interact with students in the “mock lecture”?

We have had candidates with very strong research CVs but obviously not so interested in teaching and student side of the business. On the other hand, we’ve seen extremely charismatic candidates with solid CVs leapfrog candidates that looked better on paper.

2

I don't think your example makes much sense. Your question seems to be asking how might universities evaluate research by methods other than crude citation counts, etc., whereas your example seems to be a case of not evaluating their research at all.

Of course, if you don't care about research there are many ways to evaluate candidates without using citation counts, limited only by exactly what it is that you do care about (and any applicable laws about things you're not permitted to care about).

If you do want to evaluate research more carefully than just using these metrics, then this is going to involve a more detailed look at candidates' track records, taking e.g. journal reputation into account rather than just numbers, looking at numbers and range of co-authors, getting opinions from people familiar with a candidate's research (including, but not limited to, their letter writers), and, in extremis, actually looking at the papers themselves. In fact universities already do all this, but it is much more time-consuming than looking at crude metrics, so they probably do it at a later stage after many candidates have already been eliminated (whether by doing badly on the crude but quick measures or for other reasons).

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.