32

In this forum, I am reading this great question (What is required of a mathematics referee?) by a user with the name mathprofessor. There is an answer by a user with the name Buffy which starts with:

Sorry, but if that's all you do, then your reviewing career is likely to be short, ending the first time you approve a paper that is revealed to have an error.

I am wondering now: How exactly can somebody's career as a reviewer end after some paper they reviewed is revealed to have an error?

Sure, the editor who assigned the reviewer may not assign them ever again, but how exactly are other editors (maybe from different journals) notified not to take them as a reviewer ever again? Is there some way the editor who knows is allowed to reveal the reviewer's identity? Or some higher authority they can talk to? Or how does that work out in practice?

Let us assume the following: If the answer is field-specific, let us assume we are talking about math. Moreover, as in the other question, let us assume there is no fraud going on -- the author made a honest (but big) error in the paper, and the referee was too sloppy in their report and did not note the error.

Additional question: Are there known cases where reviewers had to end their reviewing career because they did not notice an error? Again, I am assuming no fraud is going on.

Edit: I want to say that the user with name Buffy edited the answer in question and made a much weaker claim. This solves my confusion. Thank you very much, Buffy!

  • 10
    Actually, my intention was just that the editor (and that journal generally) won't come back to you. There is no pillory and no tar bucket with feathers. You just sort of disappear. And the answer was specific to math and similar things as you note. It might be different for lit criticism, though I don't know that. On the other hand, it is possible that errors are missed even with due diligence. That is why more than one reviewer is typically used. So a single error isn't necessarily review-career ending. But review in math is much more than copy editing. – Buffy Jun 6 at 21:38
  • 4
    The other question was about the actual requirements of the job of reviewing not about making a single error. The short "shock headline" quoted here doesn't capture what I actually said about the nature of the job. Please read it all before you jump to unwarranted conclusions. – Buffy Jun 7 at 0:50
  • 1
    Dear @Buffy, it's nice to hear from you and thank you for your explanation! I do, however, not understand what you mean with "please read it all". I read it all and I saw nothing more related to the "shock headline". That's why I asked. – user109595 Jun 7 at 7:03
  • 8
    @Buffy As pointed out in the comments to one of the answers below, your answer contains the phrase "your reviewing career is likely to be short, ending the first time you approve a paper that is revealed to have an error" (emphasis mine). That does indeed imply that a "single error is necessarily review-career ending". If, as you say here, your intention was just that the editor and that journal won't come back to you, and that it isn't about a single error, you may want to consider altering that sentence. – JBentley Jun 7 at 13:10
  • 9
    @Buffy Also I would point out that when it comes to written communication, the intended meaning is not anywhere near as important as the meaning most likely to be interpreted by the reader. Clearly that answer was confusing enough to generate this question by the OP, and other answers and comments seem to concur. – JBentley Jun 7 at 13:15
12

Well, not feeling obliged to review other people's papers anymore sounds like a nice deal so if you figure out an answer tell me.

Unfortunately, there won't be one. Not only do I know plenty of mathematicians who have approved papers with errors in them but I've known a number of mathematicians who were well-respected in the community despite the fact that everyone figured there was something like a 1 in 8 chance that the main claim of any paper of theirs would turn out to be fatally wrong. I can't say for certain that I've done it since obviously I would have flagged the error if I'd seen it and errors are so common that no one even mails the author for non-critical errors and a reviewer likely won't even be told if a fatal flaw in the proof is later found.

Hell, I've been halfway through extending people's published work only to email the author a question and find out that the proof is in shambles and they are struggling to find a patch. So it's literally the exact opposite situation where the total absence of errors is what would be unusual.

Indeed, I don't know anyone who has reviewed more than one or two math papers who hasn't approved a paper with an error. Studies suggest that something like 80% of published math papers contain some form of error (that's not a fatal error but still). Sorry if I don't remember the source on that study but I'm sure if you google it you can find the relevant info.


Note that I think this is a compelling reason that mathematicians should completely abandon the blind peer review process in favor of something like a math social network with up and down votes. Yes, still have two independent individuals read the paper and submit comments and demands for clarification but don't throw out all that the reviewers have learned by collapsing the judgement down to accept/reject. The mathematician I was thinking of with the frequent errors still did good work but often pursued proofs that were particularly knotty and difficult to check. The reviewers were well aware that certain parts of these proofs raised yellow flags but they couldn't specifically show there were any flaws and, since tenured professors aren't always willing to break things down to a tedious level or formality, I agree publication was the right call. However, a math social network could have passed along the reviewer's sentiment that they still have some reservations about the argument in part X moreover, the initial review will matter less since the accumulation of comments and the ability to use all professional mathematical readers of the paper as a crowdsourced continuing review will do more to help us build a mathematical edifice we are sure is true.

  • 1
    Not a mathematician but wouldn't there be a huge difference between "an error" and "a fatal error"? Like if I hear "a paper you reviewed has an error" I would expect that to mean that the error makes the claim false and that there is now no clear way to fix it, not merely that the paper contains a false statement somewhere? I guess I'm suggesting "some form of error" seems like an awfully misleading metric. – Mehrdad Jun 9 at 5:57
  • Does mathematics typically do working papers before publication? – The Thrifty Engineer Jun 9 at 17:38
57

No, the answer is simply that @buffy is wrong. In reality, editors work hard to find anyone willing to review a paper. They will be hesitant to exclude anyone. In most fields reviewers are anonymous, so only one publisher will know if a reviewer does a bad job.

  • 3
    But they won't come back to you for a second chance if you don't actually verify the mathematics to the best of your ability. And they assume that if you take on the task that you sort of agree that you have that ability. Copyediting can be done by others. It isn't the real job of the reviewer. – Buffy Jun 7 at 0:20
  • 4
    And please read the other post and my answer before you claim that "simply" I am wrong. The other post wasn't about making a single error, but about treating math reviewing as a copy editing task without really understanding the paper being reviewed. We really really want published math papers to be actually correct. The job of the reviewer is to assure that to the best of their ability. – Buffy Jun 7 at 0:40
  • 27
    You said "your reviewing career is likely to be short, ending the first time you approve a paper that is revealed to have an error" then below that you said "If you do an inadequate job for an editor, then you are likely done with that journal. Others won't know," The second statement is partly correct, and the first one contradicts the accurate part. – Anonymous Physicist Jun 7 at 1:08
25

The answer-fragment OP quoted is incorrect. Making a mistake while reviewing is not likely to lead to the end of your reviewing career. This comes from several angles (we neglect the possibility that the editors are hard-pressed to find reviewers):

  1. In most fields you can't realistically be expected to verify everything in the paper yourself. For example if you receive a paper about a new discovery at the Large Hadron Collider, you can't be expected to build your own Large Hadron Collider, run the experiments yourself, and verify the discovery. It's simply not possible.

  2. In most fields, there is some level of good faith assumed between the authors and the journal. The journal will not assume the author is actively attempting to deceive them (until proven otherwise). They will assume the author did perform the experiment. Therefore if you accept a paper that turns out to be a fraud, nobody is going to hold it against you.

  3. Finally, only the journal that you review for is likely to know your identity. No other journal will know (unless you go public). It's possible editors will tell each other not to invite a certain reviewer, or perhaps if they are sharing the same reviewer pool, but there's no central repository of "bad reviewers" or anything like that.

In practice you'll only start receiving fewer reviewer invitations (i.e. reviewing career ended) if:

  1. You retire or pass away.
  2. You make it known that you're not reviewing anymore, e.g. with a notice on your website.
  3. You become research inactive, e.g. by not publishing new papers for a while.
  • 7
    And none of those three options guarantees you won't. Indeed, I doubt even hitting all three isn't enough to stop reviewer invitations. – Peter Gerdes Jun 7 at 0:13
  • 3
    That's why I wrote "fewer reviewer invitations" =) – Allure Jun 7 at 0:17
  • 3
    Please. The other question wasn't about making an error. It was about taking a casual attitude toward the job. Reviewing in mathematics is not copy editing. If you take a casual attitude about the job then you won't be invited back to that journal. Same for the next journal who is unlucky enough to send you a paper. This question is not the same as the earlier one. – Buffy Jun 7 at 0:18
  • 6
    Shit. Guess I just killed my stack exchange reviewer career since that was a clear mistake on my part :-) – Peter Gerdes Jun 7 at 0:21
  • 1
    @PeterGerdes, bad papers are often easy to recognize. Really good ones also. It is the in between ones that are hard. Maybe they say something useful and maybe not, but the devil is in the mathematics. But if someone isn't willing to actually do the math then they shouldn't review math papers. It explains, sadly, the problems you mention in your answer here. – Buffy Jun 7 at 0:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy