5

I am wondering if some postdoc positions are evaluated using a rubric, like when students hand in tests and the instructors grade using a rubric? If so, is whoever gets the highest score the person who gets hired? In case rubrics are used, are they publicly availabe?

3
  • 10
    This totally depends on the position, there’s no standard way to grade them.
    – user438383
    Commented Jan 14, 2023 at 12:19
  • Can you give a specific country? Commented Jan 15, 2023 at 13:56
  • 3
    I voted to re-open because, while the answer to the question varies across countries, it is easily possible to give helpful comments! I was quite interested to see @FedericoPoloni's explanation of what happens in Italy. And, for the U.S., I think it's reasonable for people outside of the decision-making "elite" to wonder how decisions are made... especially with regard to issues about avoiding bias, etc. Commented Jan 17, 2023 at 0:08

7 Answers 7

15

In some countries, yes. This is for a position in Italy I am involved with (translated from Italian with the help of Google Translate):

The evaluation takes place through the attribution of a score, up to a maximum of 100 points, broken down as follows:

⦁ up to 10 points for a research doctorate or specialization diploma in the medical area;

⦁ up to 40 points for the following documented qualifications: degree mark, post-graduate diplomas, any contracts, scholarships, enrollment in doctoral or specialization schools, conference talks, other documented titles;

⦁ up to 40 points for publications and other research products;

⦁ up to 10 points for the possible interview.

The hiring committee first has to write a more detailed outline of how these items will be scored, before looking at the applications, and then proceeds to evaluate the candidates according to these criteria. There is still some space for informed judgement of the committee though; for instance the expanded rubric for the third item is:

The score is awarded considering publications that are relevant with the research program and congruent with the scientific area of the call: based on their number (up to one maximum of 10 points); on their originality, innovativeness and methodological rigor (up to a maximum of 20 points); on the scientific relevance of their publication venue and their dissemination within the scientific community (up to a maximum of 10 points).

7
  • 1
    @Buffy Yes, the winner must be the highest-scoring candidate. Commented Jan 15, 2023 at 8:54
  • 1
    @Buffy Once the scoring is done, there’s no room for rejection (if I recall correctly, it’s by law). Actually, even with this system, it’s not uncommon to have recourses in front of a court where candidates declare to have been unfairly assessed, and sometimes the selection has to be redone. In any case, no selection committee in their right mind would decide to deviate from the score. In certain cases, to avoid any recourse selection committees make very detailed rubrics. Commented Jan 15, 2023 at 9:26
  • 6
    Side note: I am not a fan of this method either; it is the product of a system where malpractice and litigation were rampant, and bureaucracy has been introduced as a complicated method to prevent them. Commented Jan 15, 2023 at 17:12
  • 3
    @Buffy I've also explained the idiosyncrasies of this process here: academia.stackexchange.com/a/59771/20058 As I jokingly said, you might not be able to hire a Nobel prize if the rubric doesn't foresee such an applicant. Personally, I think that such a system introduces more issues than those it's supposed to solve. On the other side, I had a few PhD students hired by a large US institution and the hiring process was totally devoid of bureaucracy and went about like this: hey Massimo, what is X doing? Can I offer them a position? I wish we have this freedom too. Commented Jan 15, 2023 at 21:04
  • 3
    In Germany this is not mandatory, but I know some institutions that use scoring sheets particularly if they are afraid some applicants want to sue their way into a permanent position. I.e., this is used mainly to produce a paper trail that can be used in court as evidence against discrimination allegations (in Germany, the [potential] employer has to prove non-discrimination). This also includes conducting standardized interviews. The actual questions/scored characteristics are put together according to the actual job offered, i.e., the research project's requirements. It will not be revealed Commented Jan 15, 2023 at 21:19
11

Despite the fact that they are sometimes called "trainees", from an administration perspective postdocs are generally managed as employees rather than as students.

This means that you should think of a postdoc application like a job application, rather than like any of the program admissions that you may have faced before. Critically, this means that despite the conflation of "applying to a program" vs. "applying to a laboratory" in many discussions, the hiring process is usually completely different:

  • Graduate students typically apply to a department, which handles all applications together via an admissions committee. Professors may be able to "earmark" a student by saying "I want that student / I will support that student", but ultimately it is the department and not the professor that admits the student.

  • For most postdocs, the professor is the hiring manager and has complete discretion on how their make the decision as long as it is within the institution's employment regulations.

There are exceptions to these generalizations, such as organizational fellowships, but overall you should expect there to be absolutely no uniformity across institutions or even within a single institution as to how postdoc job applications are handled.

7

I can only speak to how I evaluate candidates at a US national lab. The posting (any posting) will have a set of Required and Desired attributes. HR screens for the Required, and I generally ignore applicants that don't meet those (although I do scan to see if HR messed up - I've learned to make the Required ones as easy to understand as I can, so no technical terms).

The Desired attributes are for me to score, and I usually have 5 or so. I do it very simply - a 1 is "does not meet", a 2 is "perhaps meets", and a 3 is "definitely meets". Add those up, and one typically get a few general groupings. There are quite often a number of 'does not meet any desired attributes', some with 1 or 2, and then those with 4-5 or so. Normally there is one "high point", but some strong candidates close behind. I'm looking for the top 3 or so, if things are close I'll look at a few more but usually will only select 3. Then I'll set up interviews for about 3 candidates. The choice to hire is based solely on the interview results - the "rubric" was only to get to the interview pool. The 'high point' candidate is not always the selection.

4

I think that is unlikely in the US, at least. Much more likely is a meeting of interested faculty (one or more) who hash out the merits of candidates. They might use something like a rubric to make sure that all factors are considered, but it is really individuals making decisions about who might be best to work with and such.

I'd also guess it is impossible to create a valid rubric to be used as you suggest. There are too many variables and they have varying importance.

I can't rule out that it has happened, though.


Since the question was changed to "sometimes" I'll add the following. Yes, everything happens "sometimes". But in the US I think that faculties would consider a rubric (such as the Italian model) to be overly limiting. The problem is that you can't take advantage of opportunities that arise only after the call is published and applicants appear in the pool.

One of my (fondest) memories is that of a bunch of senior (male) mathematicians deferring to a junior (female) faculty member in the choice of a postdoc. Just by chance, an applicant turned out to be especially well qualified as a collaborator of the junior colleague and their work together could be expected to benefit both. This would, in particular, aid her in gaining tenure. But that application was serendipity that they were able to take advantage of because they weren't overly bound (and were a collegial bunch).

In general, different people have different characteristics and it is IMO not possible to build a rubric that sufficiently covers the ground unless flexibility is also possible.

2
  • 2
    Oh, ok. I thought it would be something like: grade the research statement out of 5, take the number of published papers and multiply by a certain number depending on the prestige of the journal...
    – cgb5436
    Commented Jan 14, 2023 at 12:35
  • 1
    Seems far fetched, actually. Judgement is more important in such decisions.
    – Buffy
    Commented Jan 14, 2023 at 12:44
2

There's no general answer for this because it varies immensely between countries, fields, and individual research groups––and sometimes even within them!

How hiring works generally

At one extreme, some positions will put out a call for applications, collect a bunch of them, and have them reviewed by a committee. I've never heard of a process quite as systematic as the Italian one described in Federico Poloni's answer, but it would not be odd for them to use some sort of rubric or checklist to assess candidates.

At the opposite extreme, some postdocs are arranged without much formal process. An applicant contacts a potential PI and expresses interest in joining their group. If the PI reciprocates that interest (and has money, space, etc), then a position is created and that person is hired. The applicant might be invited to give a talk or meet with other group members to help the PI make a decision, but there is no structured process and the only "competition" against opportunity costs (e.g., Will a better candidate appear in the near future?) There may be some minimal bureaucratic component, like checking that the person has a PhD, or posting a job ad, but it is almost entirely pro forma.

Crucially, both can even happen in the same department. My PhD department participated in a School of Medicine-wide postdoctoral program that was reviewed by a committee, but many postdocs were hired via informal channels.

How to get yourself hired

I'd guess you want rubrics so you can maximize your own chances of getting hired.

Unfortunately, there's no magic formula and even the same lab may have different priorities at different times: funding with tight timelines and specific deliverables often need someone who knows the field/techniques already; more open-ended funding like an NIH R01 might instead be good for someone promising who wants to learn.

In general though, all the usual markers of academic success help (papers, presentations, fellowships, etc). Interactions with the PI—either directly or via strong letters—can also make the case that you've got knowledge and skills that may not be reflected on your CV. Asking trusted mentors (your thesis committee?) may give you a rough assessment of your chances.

Another way to give your applications (of all stripes) a leg up is to clearly explain what you hope to bring to this particular lab and what you hope to get out of your time there. I think this is often neglected, but is also completely under your control. Good luck!

1

In the U.S., in math, I have never seen anything completely determined by "rubrics"... despite a long tradition of trying to make decisions by algorithms based on various numbers. :)

In math, we mostly do not have "labs", and funding is more centralized, etc., so there is less reason to give final decision-making power to individual faculty.

And, yes, in recent years, there has been a somewhat-reasonable push to make hiring and admissions fairer by having some quasi-objective rubrics play a larger role, as opposed to old-boy-network criteria. A fundamental problem is that "to compare" we seem to need "numbers", and a valid way of assigning numbers ... and adding them up... ?!?!? ... is very difficult, if not impossible. As in "really, can we quantify the prestige of journals?". And what weight to give...?!?

On another hand, the hiring/admission processes in math in the U.S. are less closed-to-outsiders than they were years ago, I think. Rubrics may help a bit, but I think even more important is a change in attitude/culture, realizing that the old almost-purely-networked culture had many implicit biases.

7
  • 1
    It seems like quantity of papers is very important. I was looking at Robert Steinberg's collected works and I don't think Steinberg would be able to get a postdoc at most places in the U.S.
    – cgb5436
    Commented Jan 15, 2023 at 23:26
  • At the risk of sounding elitist, I surmise that second-rate mathematicians value quantity of papers more and will select for postdocs who will in turn become second-rate mathematicians.
    – cgb5436
    Commented Jan 15, 2023 at 23:33
  • True, number of papers is a number. And, somewhat more subtly, some people will hire postdocs who don't make them feel threatened... All sorts of nonsense is possible, and hard to avoid by pseudo-objective rules. Yes, even not pathologically, not-so-good mathematicians may understand lukewarm mathematics better than bleeding-edge stuff, etc. As in many contexts... Commented Jan 16, 2023 at 3:32
  • Do you think academia is a meritocracy? I have heard that opinion from a senior mathematician. Doesn't it depend on how merit is measured? Merit is a social construct.
    – cgb5436
    Commented Jan 16, 2023 at 8:26
  • Certainly the pretense is that academia is a meritocracy. But, yes, there is a significant self-referential aspect to that, in addition to the "constructed" aspect (which, in may, would be denied by many, as a matter of principle :) Commented Jan 16, 2023 at 18:27
-1

In simple words, it is not necessary. However, it seems that the use of rubrics is becoming increasingly popular and is receiving unprecedented attention.

See here.

Even, the Virginia Tech Presidential Postdoctoral Fellowship RESEARCH FRONTIERS & CYBERSECURITY Proposed Scoring using Rubric. See here.

1
  • Wow! That's interesting.
    – cgb5436
    Commented Jan 15, 2023 at 22:07

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .