27

I've submitted more papers for review than I've reviewed myself. I try to review as much as I can, but it's limited by incoming requests. Publons says I'm somewhat above the median (of 0.3 reviews : 1 publication), although this data is incomplete.

My Publons review:publication ratio of 0.6:1 vs. the median of 0.3:1

It seems there must be people out there who are reviewing far more papers than they're submitting for review. But who?

Question: Who are the people reviewing far more papers than they're submitting for review?

I'm just after some mental picture of who is receiving, accepting, and completing large numbers of review requests. I can't imagine it's high-level professors, who seem constantly busy. Nor do I expect it's early-career researchers, because they're unestablished and unlikely to be thought of when editors email reviewers.

  • 16
    Anyhow, I am deeply skeptical of the Publons data. – Nate Eldredge Sep 15 at 2:59
  • 28
    You can sort Publons by number of reviews. Based on this, the answer would seem to be people like Swapnil Fegade, of the University of North Dakota Grand Forks, who if we believe that data has done 1421 reviews, 441 in the last 12 months, but never published anything. Again, I'm skeptical. – Nate Eldredge Sep 15 at 3:11
  • 9
    "Nor do I expect it's early-career researchers" -- I reviewed many more papers when I was young. From my experience, lots of reviews are from 'newbies' or newly minted PhDs, which can be frustrating at times. Nowadays, I only accept a review request if it's from an editor I know or if a paper seems to address a problem in one of my projects (very rare). As an approximation, I used to do 3-5 papers a month; nowadays, I do 5-10 papers per year, tops! – Prof. Santa Claus Sep 15 at 6:56
  • 6
    @NateEldredge Huh. I find the idea that someone is doing any reviews if they've never published to be weird at multiple levels. First, how do editors decide they are someone worth sending anything to (or even know their field)? Second, why would any be inclined to trust the reviews of someone who has never themselves done something worth publishing? That seems really strange, and makes me inclined to also be skeptical of these numbers. – JoshuaZ Sep 15 at 12:40
  • 12
    @NateEldredge many people (myself included) do not bother to import their publications into Publons. So having 0 publications is not an indicator of anything. While I am deeply sceptical of some really high review counts, it's possible that's actually a quality indicator too. A couple of my recent reviews, when the editor sent out the response to authors, I could see that at least one of the other 'reviews' was a single paragraph with almost no content. – JenB Sep 15 at 13:53
40

I can't imagine it's high-level professors, who seem constantly busy.

If you want something done, ask someone who's constantly busy.

Nor do I expect it's early-career researchers, because they're unestablished and unlikely to be thought of when editors email reviewers.

I can't point to specific posts, but I've seen enough references on this site to the scenario that I believe it's not uncommon for the editor to ask the high-level professor and the professor to delegate to an early-career researcher in their department.

  • 1
    Some actual data points: this happened to me twice, with different professors. Another time, I was asked to co-referee. – tomasz Sep 15 at 13:40
  • 10
    Note the prof usually replies to the editor to ask that the review job be sent to the postdoc directly. Of course, the prof delegating their own reviews to postdocs under the profs name would be highly unethical and definitely against most journal policies – WetlabStudent Sep 15 at 13:43
  • 8
    @WetlabStudent: still it is something not at all unheard of. – cbeleites supports Monica Sep 15 at 19:04
  • 1
    @tomasz It happened to me regularly while working in academia, possibly close to half the reviews I've done (one professor used to tell me that he had recommended to ask me, one was himself an editor who probably sent me reviews and may have recommended me in addition without telling me, one was delegating his own reviews, sometimes only asking for partial reviews on a particular aspect of the paper). I've certainly reviewed far more papers than I've submitted as first author. – cbeleites supports Monica Sep 15 at 19:09
  • 1
    @WGroleau: Well, I can't imagine this sort of thing happening in my area (what with it being so small that it would get out and ruin someone's reputation real fast). The closest thing I am aware of is some notes on a professor's website, attributed (jointly) to another one, who claims he did not contribute to them. – tomasz Sep 16 at 1:02
34

First, that ratio is based on published papers, not submitted. But the principle is the same. I was given advice several years ago that you should be reviewing 3 or 4 times as many papers as you submit.

Think about it this way; if papers need 3 reviewers and half of the papers don't get published, then the break even point is 6 reviewers per published paper. If you are in a field where 3 authors is typical, then all authors would need to review 2 papers per paper. But not all authors are able to review equivalently, so 3-4 seems to work for my discipline.

An editor would be able to give you the best information about who they ask, but senior postdocs and low to mid level academics seem to be the ones reviewing in my discipline. On the other hand, I checked publons and the highest count is actually a semi-retired professor.

  • I agree to this in everything except the number: I typically feel that reviewing twice as much as I submit is sufficient. I'm not picking up the slack for anybody else. That said, I usually have very low criteria for accepting the review until I hit the twice as many as submissions mark, but then still continue accepting reviews, just being more selective on topic match and journal quality, so I do likely end up reviewing 3-4 times as much as I submit in the end (even when "more selective" I realize I rarely have the heart to decline a review). – penelope Sep 16 at 13:00
  • My rule of thumb is: if I submit to a journal that requires x referee reports, I should referee at least x times for this journal for every paper I submit there. – ZeroTheHero Sep 16 at 22:12
28

People who submit good reviews are asked to more of them, and then more, and more, and....

Let me tell you about my own refereeing history. My research profile is not stellar, but it is adequate for a tenured theoretical physicist at a middling R1 institution; I typically publish something like two to five papers per year. On the other hand, I referee something like ten times that number of papers, and the refereeing work ends up being a major component of my professional service. In the last couple of years, I have won refereeing awards from several major journals.

Over four years as a post-doc, I got one paper per year to referee. Around the time I was established enough to get a tenure-track job, the number jumped to about one paper per month, and it has continued to increase (roughly linearly) in the decade-plus since. Unlike some of my colleagues, I try to referee practically everything I am sent that I am qualified to evaluate, and as I have built up a relationship with some journal editors, I have both been sent more papers and have been assigned to adjudicate more complicated situations, such as cases where previous referees have disagreed or appeals submitted by the authors of rejected papers.

Frankly, this does cut into some time that I could probably be usefully for research. On the other hand, I feel that, in a certain sense, I owe the professional community my expertise; I want to do at least my fair share. I work in a somewhat niche area, and there are not a lot of people who are able to give complete evaluations of research in this area. That means both making sure that papers with fundamental mistakes are not erroneously published, and making sure that the good papers are properly vetted and corrected, to maximize their usefulness. Recognizing the unacceptably bad papers quickly is a skill, but once the serious problems with a manuscript have been identified, writing a rejection recommendation is not too time consuming. Checking through good papers for minor elisions and errors, on the other hand, can take quite a bit of time, and that is where there is probably a real tradeoff between refereeing work and research output. However, as I said, I am happy with where I am currently situated in this regard.

  • 9
    Thank you for your conscientious work and contribution to the community! – user2705196 Sep 16 at 12:41
12

Authors in developed countries tend to review much more papers than they submit, while the three biggest countries for which this is the reverse are China, India, and Iran.

Source

10

Before retiring I worked for a company that (inter alia) did paid research for industrial companies. We had little incentive to publish in the peer refereed literature. There was some incentive to publish in the trade literature, but not much - we were very well known among potential clients. We did participate in industry and regulatory groups as contributions to the "greater good" for the industry. I and many of my senior colleagues reviewed many more papers than we submitted; I can't speak for my colleagues but I regarded it again as contributing to the "greater good".
Edited to add:
Similarly, I had no interest at all in recording my contributions in Publons, which Wikipedia describes as (my emphasis) "...service for academics to track, verify, and showcase their peer review and editorial contributions...".

6

I suspect a major factor is that ratios will vary significantly by field. In my field (pure math), a typical paper has 1-3 authors and will get reviews from two people (more if it gets rejected post-review and sent to another journal, although overlap of reviewers in that case is not uncommon). So I'd expect the median ratio there to be somewhere around 1. Mine is 1.3 FWIW (I am not on publons but keep records for my own interest).

In other fields the typical number of authors per paper could be much higher without the number of reviewers per paper significantly increasing, so I would expect a much smaller ratio. 0.3 seems reasonable for fields where 10+ author papers are the norm.

  • 2
    +1 for pointing out that the average number of reviews per person will be less than the average number of papers per person if the average number of authors/paper is greater than the average number of reviewers/paper. – Kimball Sep 16 at 16:10
  • (I also think a ratio of around 1 is good for pure math.) – Kimball Sep 16 at 16:10
  • A comment on the question suggests Publons counts reviewing one paper through two rounds as two reviews, not one. In that case my ratio would be significantly higher (about 2). – Especially Lime Sep 17 at 10:00
6

Actually the most reviewing is, in my field and related ones, done exactly by the two categories that you wanted dismiss.

A reputed professor receive a large number of invitations, so that the ratio is high in spite of refusals.

A successful and young researcher receive relatively few but s/he tends to accept all as it is a new and certainly formative experience.

I suspect, as another answer points out, it's very much field and subfield related. I do personally reviewed much more papers than those I've published.

While a researcher in synthetic chemistry publish and might be invited by a few good journals, one in material science can do that in a broader range, spanning from physics and chemistry to dedicated publications. When preparation of new materials is involved, basically all papers are getting interesting and reviewing becomes a way to stay tune.

Similar scenario is plausible in many applied sciences.

In my field the number of reviewings largely exceeds that of the submitted/published papers for literally every graduate researcher that I ever met.

Also a ratio as those given in your Q, if not individually calculated, would block research publications. Simple maths suggests that ratio lower than one would be a big problem in the current procedure, except perhaps in fields where having about ten authors or more is common (still some of those fields have also more abundant literature, e.g. medicine). Also having the name in a author list doesn't make you a referree, at least not one of first and even second choice for a good journal.

Edit 1 after the comment.

Edit 2: Stimulated by this I've finally registered on Publon. While importing most of my papers was a simple click, the thousand review I have done in the past will never show up. So at the current stage I have contributed X papers to literature without having been referee at all. I've overlooked the fact that OP inspiration was Publon. Basically she can be confident of how much she is reviewing (not enough in my opinion, as for what I and others have written above) but in general one should be highly skeptical of those data, independent of their actual significance.

  • "Also a ratio as those given in your Q would block research publications. It is simple maths." It'd depend on what the curve looks like - even if the bulk of reviewers are at .3 papers reviewed / paper submitted, a smaller proportion of reviewers with a significantly higher ratio would be able to life the overall propertion to 1:1. – nick012000 Sep 16 at 11:39
  • @nick012000 I was aware of such a comment as for one can play with numbers. First of all a small proportion of heavy reviewers lift up the average anyway. Assuming equally distributed load the ratio should be at least above 1. In some fields there are 10+ authors, but in many 1 is common. Referees are normally at least two. And many journals claim very very high rejection rates. I wanted to convey a message and not demonstrate something. I'll edit. Thanks. – Alchimista Sep 16 at 12:11
  • 2
    Since it's the median they quoted, not the mean, the quoted figure would be less susceptible to being distorted by a small number of very large values. – nick012000 Sep 16 at 12:16
5

I am one of your mystery reviewers. I work in research in industry, and I have gotten a positive reputation in my field for my work. As such I am frequently requested to review journal articles and since my combination of fields is relatively rare, I generally feel a duty to help ensure the quality of the contributions in that field through reviewing articles (which I frequently do on my own time).

On the other hand, submitting articles is more of a secondary (or even tertiary) function of my job. Any publications have to be cleared through our IP team to determine if we need to get a patent filed first or even try to protect the knowledge as a trade secret. When I do publish, it is usually to establish the value of a product or it is part of an academic collaboration I have been engaged in. As such, I only publish about once a year to every other year.

4

This is just a theory: The most active reviewers are editors of low-ranked journals, who assign the reviews to themselves.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.