Why do errors exist in peer reviewed publications?

Question

I found a mistake (not a misprint) in an article which has been published in a very good math journal. According the available data, the article was in peer-review process for more than 8 months. But I do not know why the referee(s) did not notice that. What are reasons for this kind of articles?

Answer 1

People make mistakes.

Manuscript authors, reviewers, and editors are people, and people are not perfect. Even if every person involved in the publication of a manuscript catches 99% of all errors, it's still possible that some errors will go unnoticed. This likelihood of course goes up when authors/reviewers are careless, but it can never be eliminated entirely even with very meticulous review.

Answer 2

Going by my experience in reviewing papers, it's virtually impossible to really review every aspect of a paper. This is particularly the case in some fields where the supplementary methods go for 100s of pages and the code may be 10s of 1000s of lines long. It's unlikely that all reviewers have the in depth knowledge of specific fields and the huge amount of time required to properly evaluate everything in the paper. This is particularly the case in fields which are relatively new and ground-breaking - there simply doesn't exist the pool of capable reviewers.

For example, I recently reviewed a paper for a top journal (Nature Genetics) which involved a complex and quite a specialised piece of software for genetic analysis. There was no way I could evaluate each line of code they had written, so there is no way the reviewer could be expected to detect bugs and errors in the code. We have to look at the results presented and their justification to then assume that the code is working as the authors said it would. There is a rather large amount of faith in the authors that their code does what they say it does. I could have spent a whole month non stop reviewing the paper and I still imagine minor things could have slipped by.

This issue is compounded by a) the fact reviewers don't get paid and b) reviewers are often very overworked and c) there isn't much motivation beyond professionalism and academic ideals (although of course these can be strong motivating factors) to spend a large amount of time meticulously reviewing papers.

tl;dr. Reviewers mostly do the best job they can, but given the terrible structure set in place to perform most peer-reviews, I certainly wouldn't trust a peer-review to be a cast iron certificate that everything in the paper is correct, but fortunately a vast majority of people in academia are honest and don’t try to deceive readers or reviewers.

Answer 3

Ideally, this would not happen, but it is near unavoidable.

For instance, if you are refereeing a proof, chances are you won't read it line-by-line. In fact, when I can only read a mathematical argument that way, it is because I have not yet understood it, and this is a really bad way of verifying global correctness, or coherence, or originality. A lot may be lost, and even if everything is correct, chances are that at the end it is hard to say with any certainty how flexible the argument is, whether all the assumptions are truly needed, whether some steps are really as difficult as depicted, or why. Usually we read modularly, trying to grasp the global structure of the argument, using lemmas and the like as black boxes. Only when that structure makes sense, we proceed to the lemmas. Or we may not even read the proofs of the lemmas, because we see how to prove them ourselves. Now, if one such proof has a problem as written in the paper, there is a chance we will miss it because we see how to prove the lemma anyway. Some referees do not care much about mistakes at this level, since they are easily fixable, and what matters more is that the overall argument is sturdy.

It would be much better, overall, if papers included extended discussions of motivation, intuition, proof strategies, and so on. People reading them would find the arguments easier to digest, and the possibility of missing a mistake would reduce. But technical writing is difficult to begin with, some journals have page limits, and sometimes there are time constraints (related to tenure or promotion considerations, for example) that restrict the ability of authors to spend the time that this inclusion would require. Of course, not having such remarks throughout the paper may make it difficult for referees and other readers to grasp some of the details, which may lead to missing mistakes.

Ideally, we as referees should read the paper several times, at least once line-by-line, but few times we are in the position to devote so much time to the process. When I can afford the time, I may even comment on typos or style, though I much prefer if the substance of my comments is on the mathematics of the paper and its potential for generalizations or extensions, or connections with other works. A few happy times I've seen how to improve some of the proofs presented in the paper, but I imagine I have missed important details as well.

Of course, it may well be that a paper has an error that is not at the level of a typo or a lemma not quite proved as it should be. An error may be significant and we may still miss it. Sometimes we find an argument similar to something we are familiar with, and skip verifying details we expect to be routine, and end up missing something serious. Or we misunderstand. It is really not that uncommon or surprising.

Papers are not written in formal languages that are machine-verifiable. Some people argue that they should be. Whatever is the case, currently most of our proofs are conversational, and technicalities may be omitted on occasion. Many papers are very dense, and it takes years of careful examination by many people to detect flaws or gaps, or genuine mistakes. Peer review is not meant to signify a perfect guarantee of correctness, and it is a mistake to think of it as having that goal.

Here is a quote regarding the refereeing process of the journal Discrete Analysis, highlighting precisely this last point:

In some cases, it is not reasonable to expect a reviewer to check the correctness of a paper down to the last detail. In such cases, editors may be satisfied with indirect evidence that a paper is likely to be correct. (For example, it may be that the general outline of the argument is convincing, but that the technical details involved in converting the outline into a complete proof are very complicated.) Thus, publication in Discrete Analysis should not be considered an absolute guarantee of correctness, just as in practice it is not a guarantee for any other journal.

Answer 4

I'm an engineer and peer reviewing papers in my field for complete correctness is nearly impossible, as papers typically summarize findings with novel computer codes or experiments I don't have direct access to. I review a paper to see that it is free of obvious fallacy, on topic for the publication, and appears to represent work of sufficient quality and novelty that subsequent researchers might find it useful. If I'm satisfied on these counts, I'll click "recommend for publication". I'll leave comments on minor issues if I see them, but I don't see it as my job to proofread or edit papers for free.

Engineers generate a lot of data and models on obscure topics, so I suspect these standards are a bit different than in some top scientific journals. However, I would encourage anyone who doesn't have first hand experience reviewing papers to view peer review as more of a screening process to weed out trash than make sure what makes it to print is absolutely correct.

Answer 5

I believe that human error is the most common reason for inaccuracy in peer-reviewed scientific articles, but I must respectfully add another reason: academic politics. I served as a peer reviewer for over 20 years. I typically spent 10-20 hours on each review, because informational integrity is the bedrock of scientific advancement.

The longer I served as a peer reviewer, the more frequently I received articles on controversial topics or from "big names" in my field. Reviews are supposed to be anonymous, but when authors refer to "our previous work (xxx, 2012)" anonymity disappears.

Sadly, it became apparent that some senior Editors were less willing to require revisions from "big names" who used inferior methods, incorrectly interpreted the meaning of their statistical results, or who badly mischaracterized the results of prior research. Errors that would have sounded the death knell for a paper written by a newbie were overlooked, dismissed, and excused. (My own research became increasingly arduous as I realized that I needed to check the veracity of any citation I encountered before relying upon it.)

I stopped peer reviewing after spending three weeks on a particularly critical article by a "big name". This person had used a type of analysis with which I was familiar, but which was not well known in my field. Quite simply, the paper was a train wreck. Among several other serious errors: Basic tenets of the theory underlying the analysis were ignored, misstated, or inaccurately cited. Mathematical calculations were just flat out incorrect. And rather than using standard measurement instruments common in the field, the author used untested, unpublished instruments developed in his/her lab.

I was not a "big name," and I wanted to make certain that I was on very solid ground in my critique. I documented every single concern in my eight-page review with references to published material; never did I leave the senior Editor to just "trust my judgment." Where I knew that the senior Editor might have difficulty tracking down a source, I sent photocopies of the material I had used.

The article was accepted for publication without a single substantive change. The other reviewers, who admitted that they knew nothing about the fairly dense theoretical basis of the analysis, returned half-page reviews that mentioned only typographical errors and run-on sentences. The senior Editor was not willing to discuss the decision with me.

As I said in my opening statement, I do not believe that academic politics accounts for most of the errors in peer-reviewed research. I think that most stem from honest human error or from unfortunate carelessness. I post this response to round out the answer to the OP, and to make explicit the fact that "big names" do not guarantee "good science" or error-free publications.

Answer 6

The reviewer is not responsible for the content of an article. He gives his opinion on the novelty and scientific rigour, and recommends acceptance or rejection based upon that opinion. He has the freedom to recommend improvements, and is expected to point out whatever appears to be erroneous to him.

The author must not lie (that would create a liability on his side), but of course can err like all humans do in nearly every waking moment.

The reader however must never take anything for granted just because it was written down in an article. It is the reader who must double-check everything and still carry the full responsibility if he looses time, money, or a limb based on a misconception that was transported in some scientific article.

Answer 7

I would say there is another source of errors in peer-reviewed publications. The latter need to contain something new, so they are often state-of-the-art, and it is just hard both to produce and review their results, hence the errors.