14

I found a mistake (not a misprint) in an article which has been published in a very good math journal. According the available data, the article was in peer-review process for more than 8 months. But I do not know why the referee(s) did not notice that. What are reasons for this kind of articles?

100

People make mistakes.

Manuscript authors, reviewers, and editors are people, and people are not perfect. Even if every person involved in the publication of a manuscript catches 99% of all errors, it's still possible that some errors will go unnoticed. This likelihood of course goes up when authors/reviewers are careless, but it can never be eliminated entirely even with very meticulous review.

| improve this answer | |
  • 10
    As simple as that. – henning -- reinstate Monica Sep 16 at 14:55
  • 4
    For what it's worth (coming from the software business), that's why there's buggy software. Most software it code reviewed by one to 4 (sometime more) people, and subject to automated tests. But, it turns out that People do make mistakes, authors, reviewers, test authors, etc. As a result, cash registers restart at random, web sites have glitches and even things like Boeing planes and Arianne rockets sometimes fall out of the skies. – Flydog57 Sep 17 at 1:23
  • 4
    It's not unknown for journals to rely on a single reviewer, and it's hardly surprising if that reviewer is only sufficiently expert on most of the paper. Even multiple reviewers aren't guaranteed to cover the whole scope perfectly. While gross errors should still be caught, as well as many issues, subtle but material errors can easily slip through. – Chris H Sep 17 at 8:37
  • @ChrisH Exactly right! "Peer review" doesn't necessarily mean thoroughly checked (though it might), it just means that is was checked at least once by somebody at least theoretically competent. – Michael W. Sep 17 at 17:25
  • 1
    @henning--reinstateMonica. I am not sure it is just 'As simple as that'. Of course people make mistakes, but I suspect a much larger reason why errors occur is because the review system is totally broken, and that reviewers aren't professionalised. I think this gives the idea that errors in the peer-review system are unavoidable, rather than fundamentally caused by greedy journals . It doesn't encourage people to try and change the system if they think it is inevitable. – user438383 Sep 18 at 10:27
18

Going by my experience in reviewing papers, it's virtually impossible to really review every aspect of a paper. This is particularly the case in some fields where the supplementary methods go for 100s of pages and the code may be 10s of 1000s of lines long. It's unlikely that all reviewers have the in depth knowledge of specific fields and the huge amount of time required to properly evaluate everything in the paper. This is particularly the case in fields which are relatively new and ground-breaking - there simply doesn't exist the pool of capable reviewers.

For example, I recently reviewed a paper for a top journal (Nature Genetics) which involved a complex and quite a specialised piece of software for genetic analysis. There was no way I could evaluate each line of code they had written, so there is no way the reviewer could be expected to detect bugs and errors in the code. We have to look at the results presented and their justification to then assume that the code is working as the authors said it would. There is a rather large amount of faith in the authors that their code does what they say it does. I could have spent a whole month non stop reviewing the paper and I still imagine minor things could have slipped by.

This issue is compounded by a) the fact reviewers don't get paid and b) reviewers are often very overworked and c) there isn't much motivation beyond professionalism and academic ideals (although of course these can be strong motivating factors) to spend a large amount of time meticulously reviewing papers.

tl;dr. Reviewers mostly do the best job they can, but given the terrible structure set in place to perform most peer-reviews, I certainly wouldn't trust a peer-review to be a cast iron certificate that everything in the paper is correct, but fortunately a vast majority of people in academia are honest and don’t try to deceive readers or reviewers.

| improve this answer | |
9

Ideally, this would not happen, but it is near unavoidable.

For instance, if you are refereeing a proof, chances are you won't read it line-by-line. In fact, when I can only read a mathematical argument that way, it is because I have not yet understood it, and this is a really bad way of verifying global correctness, or coherence, or originality. A lot may be lost, and even if everything is correct, chances are that at the end it is hard to say with any certainty how flexible the argument is, whether all the assumptions are truly needed, whether some steps are really as difficult as depicted, or why. Usually we read modularly, trying to grasp the global structure of the argument, using lemmas and the like as black boxes. Only when that structure makes sense, we proceed to the lemmas. Or we may not even read the proofs of the lemmas, because we see how to prove them ourselves. Now, if one such proof has a problem as written in the paper, there is a chance we will miss it because we see how to prove the lemma anyway. Some referees do not care much about mistakes at this level, since they are easily fixable, and what matters more is that the overall argument is sturdy.

It would be much better, overall, if papers included extended discussions of motivation, intuition, proof strategies, and so on. People reading them would find the arguments easier to digest, and the possibility of missing a mistake would reduce. But technical writing is difficult to begin with, some journals have page limits, and sometimes there are time constraints (related to tenure or promotion considerations, for example) that restrict the ability of authors to spend the time that this inclusion would require. Of course, not having such remarks throughout the paper may make it difficult for referees and other readers to grasp some of the details, which may lead to missing mistakes.

Ideally, we as referees should read the paper several times, at least once line-by-line, but few times we are in the position to devote so much time to the process. When I can afford the time, I may even comment on typos or style, though I much prefer if the substance of my comments is on the mathematics of the paper and its potential for generalizations or extensions, or connections with other works. A few happy times I've seen how to improve some of the proofs presented in the paper, but I imagine I have missed important details as well.

Of course, it may well be that a paper has an error that is not at the level of a typo or a lemma not quite proved as it should be. An error may be significant and we may still miss it. Sometimes we find an argument similar to something we are familiar with, and skip verifying details we expect to be routine, and end up missing something serious. Or we misunderstand. It is really not that uncommon or surprising.

Papers are not written in formal languages that are machine-verifiable. Some people argue that they should be. Whatever is the case, currently most of our proofs are conversational, and technicalities may be omitted on occasion. Many papers are very dense, and it takes years of careful examination by many people to detect flaws or gaps, or genuine mistakes. Peer review is not meant to signify a perfect guarantee of correctness, and it is a mistake to think of it as having that goal.

Here is a quote regarding the refereeing process of the journal Discrete Analysis, highlighting precisely this last point:

In some cases, it is not reasonable to expect a reviewer to check the correctness of a paper down to the last detail. In such cases, editors may be satisfied with indirect evidence that a paper is likely to be correct. (For example, it may be that the general outline of the argument is convincing, but that the technical details involved in converting the outline into a complete proof are very complicated.) Thus, publication in Discrete Analysis should not be considered an absolute guarantee of correctness, just as in practice it is not a guarantee for any other journal.

| improve this answer | |
5

I'm an engineer and peer reviewing papers in my field for complete correctness is nearly impossible, as papers typically summarize findings with novel computer codes or experiments I don't have direct access to. I review a paper to see that it is free of obvious fallacy, on topic for the publication, and appears to represent work of sufficient quality and novelty that subsequent researchers might find it useful. If I'm satisfied on these counts, I'll click "recommend for publication". I'll leave comments on minor issues if I see them, but I don't see it as my job to proofread or edit papers for free.

Engineers generate a lot of data and models on obscure topics, so I suspect these standards are a bit different than in some top scientific journals. However, I would encourage anyone who doesn't have first hand experience reviewing papers to view peer review as more of a screening process to weed out trash than make sure what makes it to print is absolutely correct.

| improve this answer | |
  • Indeed, "a screening process". Both to weed out lower-quality papers and to maintain some notion of "status" of the journal, which is a different thing, though related. – paul garrett Sep 18 at 21:43
  • And computer code and data analysis are the parts that would be easiest to exchange. There may still be errors in the experimental setup, including decisions/choices that are not even known yet to be suboptimal or flawed. – cbeleites unhappy with SX Sep 19 at 11:09
  • Code and data are indeed easy to exchange, but they are not typically offered to reviewers. Additionally, the time required for doing code or raw data reviews is well beyond what one can reasonably expect of a professional getting neither pay or credit for service as a reviewer. – Brian d'Entremont Sep 21 at 2:06
2

I believe that human error is the most common reason for inaccuracy in peer-reviewed scientific articles, but I must respectfully add another reason: academic politics. I served as a peer reviewer for over 20 years. I typically spent 10-20 hours on each review, because informational integrity is the bedrock of scientific advancement.

The longer I served as a peer reviewer, the more frequently I received articles on controversial topics or from "big names" in my field. Reviews are supposed to be anonymous, but when authors refer to "our previous work (xxx, 2012)" anonymity disappears.

Sadly, it became apparent that some senior Editors were less willing to require revisions from "big names" who used inferior methods, incorrectly interpreted the meaning of their statistical results, or who badly mischaracterized the results of prior research. Errors that would have sounded the death knell for a paper written by a newbie were overlooked, dismissed, and excused. (My own research became increasingly arduous as I realized that I needed to check the veracity of any citation I encountered before relying upon it.)

I stopped peer reviewing after spending three weeks on a particularly critical article by a "big name". This person had used a type of analysis with which I was familiar, but which was not well known in my field. Quite simply, the paper was a train wreck. Among several other serious errors: Basic tenets of the theory underlying the analysis were ignored, misstated, or inaccurately cited. Mathematical calculations were just flat out incorrect. And rather than using standard measurement instruments common in the field, the author used untested, unpublished instruments developed in his/her lab.

I was not a "big name," and I wanted to make certain that I was on very solid ground in my critique. I documented every single concern in my eight-page review with references to published material; never did I leave the senior Editor to just "trust my judgment." Where I knew that the senior Editor might have difficulty tracking down a source, I sent photocopies of the material I had used.

The article was accepted for publication without a single substantive change. The other reviewers, who admitted that they knew nothing about the fairly dense theoretical basis of the analysis, returned half-page reviews that mentioned only typographical errors and run-on sentences. The senior Editor was not willing to discuss the decision with me.

As I said in my opening statement, I do not believe that academic politics accounts for most of the errors in peer-reviewed research. I think that most stem from honest human error or from unfortunate carelessness. I post this response to round out the answer to the OP, and to make explicit the fact that "big names" do not guarantee "good science" or error-free publications.

| improve this answer | |
New contributor
Cyclone93 is a new contributor to this site. Take care in asking for clarification, commenting, and answering. Check out our Code of Conduct.
1

The reviewer is not responsible for the content of an article. He gives his opinion on the novelty and scientific rigour, and recommends acceptance or rejection based upon that opinion. He has the freedom to recommend improvements, and is expected to point out whatever appears to be erroneous to him.

The author must not lie (that would create a liability on his side), but of course can err like all humans do in nearly every waking moment.

The reader however must never take anything for granted just because it was written down in an article. It is the reader who must double-check everything and still carry the full responsibility if he looses time, money or a limb based on a misconception that was transported in some scientific article.

| improve this answer | |
0

I would say there is another source of errors in peer-reviewed publications. The latter need to contain something new, so they are often state-of-the-art, and it is just hard both to produce and review their results, hence the errors.

| improve this answer | |

Not the answer you're looking for? Browse other questions tagged or ask your own question.