34

In this forum, I am reading this great question (What is required of a mathematics referee?) by a user with the name mathprofessor. There is an answer by a user with the name Buffy which starts with:

Sorry, but if that's all you do, then your reviewing career is likely to be short, ending the first time you approve a paper that is revealed to have an error.

I am wondering now: How exactly can somebody's career as a reviewer end after some paper they reviewed is revealed to have an error?

Sure, the editor who assigned the reviewer may not assign them ever again, but how exactly are other editors (maybe from different journals) notified not to take them as a reviewer ever again? Is there some way the editor who knows is allowed to reveal the reviewer's identity? Or some higher authority they can talk to? Or how does that work out in practice?

Let us assume the following: If the answer is field-specific, let us assume we are talking about math. Moreover, as in the other question, let us assume there is no fraud going on -- the author made a honest (but big) error in the paper, and the referee was too sloppy in their report and did not note the error.

Additional question: Are there known cases where reviewers had to end their reviewing career because they did not notice an error? Again, I am assuming no fraud is going on.

Edit: I want to say that the user with name Buffy edited the answer in question and made a much weaker claim. This solves my confusion. Thank you very much, Buffy!

Improve this question
9
  • 11
    Actually, my intention was just that the editor (and that journal generally) won't come back to you. There is no pillory and no tar bucket with feathers. You just sort of disappear. And the answer was specific to math and similar things as you note. It might be different for lit criticism, though I don't know that. On the other hand, it is possible that errors are missed even with due diligence. That is why more than one reviewer is typically used. So a single error isn't necessarily review-career ending. But review in math is much more than copy editing.
    – Buffy
    Jun 6, 2019 at 21:38
  • 4
    The other question was about the actual requirements of the job of reviewing not about making a single error. The short "shock headline" quoted here doesn't capture what I actually said about the nature of the job. Please read it all before you jump to unwarranted conclusions.
    – Buffy
    Jun 7, 2019 at 0:50
  • 2
    Dear @Buffy, it's nice to hear from you and thank you for your explanation! I do, however, not understand what you mean with "please read it all". I read it all and I saw nothing more related to the "shock headline". That's why I asked.
    – user109595
    Jun 7, 2019 at 7:03
  • 11
    @Buffy As pointed out in the comments to one of the answers below, your answer contains the phrase "your reviewing career is likely to be short, ending the first time you approve a paper that is revealed to have an error" (emphasis mine). That does indeed imply that a "single error is necessarily review-career ending". If, as you say here, your intention was just that the editor and that journal won't come back to you, and that it isn't about a single error, you may want to consider altering that sentence.
    – JBentley
    Jun 7, 2019 at 13:10
  • 11
    @Buffy Also I would point out that when it comes to written communication, the intended meaning is not anywhere near as important as the meaning most likely to be interpreted by the reader. Clearly that answer was confusing enough to generate this question by the OP, and other answers and comments seem to concur.
    – JBentley
    Jun 7, 2019 at 13:15

3 Answers 3

Reset to default
14

Well, not feeling obliged to review other people's papers anymore sounds like a nice deal so if you figure out an answer tell me.

Unfortunately, there won't be one. Not only do I know plenty of mathematicians who have approved papers with errors in them but I've known a number of mathematicians who were well-respected in the community despite the fact that everyone figured there was something like a 1 in 8 chance that the main claim of any paper of theirs would turn out to be fatally wrong. I can't say for certain that I've done it since obviously I would have flagged the error if I'd seen it and errors are so common that no one even mails the author for non-critical errors and a reviewer likely won't even be told if a fatal flaw in the proof is later found.

Hell, I've been halfway through extending people's published work only to email the author a question and find out that the proof is in shambles and they are struggling to find a patch. So it's literally the exact opposite situation where the total absence of errors is what would be unusual.

Indeed, I don't know anyone who has reviewed more than one or two math papers who hasn't approved a paper with an error. Studies suggest that something like 80% of published math papers contain some form of error (that's not a fatal error but still). Sorry if I don't remember the source on that study but I'm sure if you google it you can find the relevant info.

Note that I think this is a compelling reason that mathematicians should completely abandon the blind peer review process in favor of something like a math social network with up and down votes. Yes, still have two independent individuals read the paper and submit comments and demands for clarification but don't throw out all that the reviewers have learned by collapsing the judgement down to accept/reject. The mathematician I was thinking of with the frequent errors still did good work but often pursued proofs that were particularly knotty and difficult to check. The reviewers were well aware that certain parts of these proofs raised yellow flags but they couldn't specifically show there were any flaws and, since tenured professors aren't always willing to break things down to a tedious level or formality, I agree publication was the right call. However, a math social network could have passed along the reviewer's sentiment that they still have some reservations about the argument in part X moreover, the initial review will matter less since the accumulation of comments and the ability to use all professional mathematical readers of the paper as a crowdsourced continuing review will do more to help us build a mathematical edifice we are sure is true.

Improve this answer
3
  • 1
    Not a mathematician but wouldn't there be a huge difference between "an error" and "a fatal error"? Like if I hear "a paper you reviewed has an error" I would expect that to mean that the error makes the claim false and that there is now no clear way to fix it, not merely that the paper contains a false statement somewhere? I guess I'm suggesting "some form of error" seems like an awfully misleading metric.
    – user541686
    Jun 9, 2019 at 5:57
  • Does mathematics typically do working papers before publication?
    – The Thrifty Engineer
    Jun 9, 2019 at 17:38
  • @user541686 Well, yes there is a difference between an error and fatal error but it's less clear cut than you might think. For instance, does the proof P!=NP because "3-SAT isn't in P" contains a fatal error or non-fatal error? It's totally unenlightening but since we know P!=NP iff 3-SAT isn't in P there is a sense in which if anyone ever proves P!=NP they've plugged the hole in this proof. So in practical terms there is whether or not the theorem is true and then gaps in the proof of varying severity from trivial typo to requires decades/centuries of research to plug.
    – Peter Gerdes
    Jun 2, 2020 at 8:50
58

No, the answer is simply that @buffy is wrong. In reality, editors work hard to find anyone willing to review a paper. They will be hesitant to exclude anyone. In most fields reviewers are anonymous, so only one publisher will know if a reviewer does a bad job.

Improve this answer
3
  • 3
    But they won't come back to you for a second chance if you don't actually verify the mathematics to the best of your ability. And they assume that if you take on the task that you sort of agree that you have that ability. Copyediting can be done by others. It isn't the real job of the reviewer.
    – Buffy
    Jun 7, 2019 at 0:20
  • 4
    And please read the other post and my answer before you claim that "simply" I am wrong. The other post wasn't about making a single error, but about treating math reviewing as a copy editing task without really understanding the paper being reviewed. We really really want published math papers to be actually correct. The job of the reviewer is to assure that to the best of their ability.
    – Buffy
    Jun 7, 2019 at 0:40
  • 31
    You said "your reviewing career is likely to be short, ending the first time you approve a paper that is revealed to have an error" then below that you said "If you do an inadequate job for an editor, then you are likely done with that journal. Others won't know," The second statement is partly correct, and the first one contradicts the accurate part.
    – Anonymous Physicist
    Jun 7, 2019 at 1:08
26

The answer-fragment OP quoted is incorrect. Making a mistake while reviewing is not likely to lead to the end of your reviewing career. This comes from several angles (we neglect the possibility that the editors are hard-pressed to find reviewers):

  1. In most fields you can't realistically be expected to verify everything in the paper yourself. For example if you receive a paper about a new discovery at the Large Hadron Collider, you can't be expected to build your own Large Hadron Collider, run the experiments yourself, and verify the discovery. It's simply not possible.

  2. In most fields, there is some level of good faith assumed between the authors and the journal. The journal will not assume the author is actively attempting to deceive them (until proven otherwise). They will assume the author did perform the experiment. Therefore if you accept a paper that turns out to be a fraud, nobody is going to hold it against you.

  3. Finally, only the journal that you review for is likely to know your identity. No other journal will know (unless you go public). It's possible editors will tell each other not to invite a certain reviewer, or perhaps if they are sharing the same reviewer pool, but there's no central repository of "bad reviewers" or anything like that.

In practice you'll only start receiving fewer reviewer invitations (i.e. reviewing career ended) if:

  1. You retire or pass away.
  2. You make it known that you're not reviewing anymore, e.g. with a notice on your website.
  3. You become research inactive, e.g. by not publishing new papers for a while.
Improve this answer
15
  • 7
    And none of those three options guarantees you won't. Indeed, I doubt even hitting all three isn't enough to stop reviewer invitations.
    – Peter Gerdes
    Jun 7, 2019 at 0:13
  • 3
    That's why I wrote "fewer reviewer invitations" =)
    – Allure
    Jun 7, 2019 at 0:17
  • 3
    Please. The other question wasn't about making an error. It was about taking a casual attitude toward the job. Reviewing in mathematics is not copy editing. If you take a casual attitude about the job then you won't be invited back to that journal. Same for the next journal who is unlucky enough to send you a paper. This question is not the same as the earlier one.
    – Buffy
    Jun 7, 2019 at 0:18
  • 6
    Shit. Guess I just killed my stack exchange reviewer career since that was a clear mistake on my part :-)
    – Peter Gerdes
    Jun 7, 2019 at 0:21
  • 1
    @PeterGerdes, bad papers are often easy to recognize. Really good ones also. It is the in between ones that are hard. Maybe they say something useful and maybe not, but the devil is in the mathematics. But if someone isn't willing to actually do the math then they shouldn't review math papers. It explains, sadly, the problems you mention in your answer here.
    – Buffy
    Jun 7, 2019 at 0:36

You must log in to answer this question.