How often do people find mistakes in papers and not tell anyone?

Question

I found a significant error in a paper with 1000 citations. My advisor agreed that it is a significant error, but discouraged me from telling the author.

It occurred to me that maybe other graduate students also found the error, and never told the author, or they did tell the author and he just never corrected it. I think an author would have very little incentive to correct his own work, especially if they were famous for that work, and readers are strongly incentivized against writing a correction paper, because the author would be mad at them.

How often does this sort of thing happen? Can we really trust the literature?

Answer 1

I'm not sure what you are looking for. I know for sure that the seminal paper that made one of my bosses famous had glaring mistakes in it. Despite the fact that his result was completely wrong, the paper started a new research field and is now highly cited (around 1700, I think). No one ever bothered to point out the calculation mistakes, without which the paper would have never gotten accepted, because his proposal got confirmed experimentally. More detailed calculations done by others showed his mistake, but confirmed his intuition. In any case, I can't find survey data in my field dealing with un(der)reported mistakes in papers. But, as you do research, you are bound to find quite a few.

On the other hand, I found this oncology paper on unreported mistakes in oncology papers http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3031354/

Looks like less than 25% of those who spotted mistakes in the papers, actually went on to report them.

The practice in my field is to write the authors about the possible mistake. If they are willing to correct it, there is not much point in escalating. If not, you can write a comment on how wrong the paper was, post it on arxiv and send it to the journal editor. This, assuming you're sure they made a mistake. Most people aren't willing to go through all this pain, but I've seen this many times.

Answer 2

Researchers/authors are human.

Humans make errors.

Thus Researchers/authors make errors (if we assume transitivity is given).

So, can we trust literature? Well, being an agnostic I tend to never blindly trust any statement. I found dozens of errors in the lectures I attended and made the experience that lecturers respond very different to criticism. There are some that feel attacked, or those that simply do not bother, but there are also those that will really think about your criticism and will correct you or themselves after having carefully checked the issue in question again.

I think, that a researcher's duty should be to allow and work with criticism as this will lead to an even better understanding of the topic for all involved sides. Also, your criticism shows interest which is actually a good thing and should be rated as such.

A good criticism should focus on the research itself and leave out personal-related stuff.

So my conclusion is, that you shouldn't just do nothing because of those strange unspoken rules society has built up to underline the higher status of academic people. Remaining silent for social fears is a bad thing that leads to a vicious cycle and harms research in my opinion. You should be free to question whenever you feel like it is necessary to question. Consider for yourself for when it makes sense to consume someone else's time. If you are not sure about your criticism, make some own research. If you are sure your criticism is right, if you ask me, for research itself this is the best one can do.

Answer 3

Slightly tangential (doesn't answer "can we trust the literature?"), but:

On several occasions, especially when working with a class of students to work through a paper line-by-line and replicate its results, I/we have found errors — nothing that changes the qualitative conclusions of the paper, but definitely large enough to be confusing to students. These kinds of errors are overlooked by most readers because they aren't going through the paper as thoroughly, but setting the record straight can save a lot of time and trouble for future readers — especially students, who won't have the courage of their convictions and will think that they must have made a mistake/be misunderstanding something.

It helps of course that (1) I was already reasonably well established in the field and knew/was known to the authors and (2) the authors were reasonable people who saw the value in correcting their minor (but confusing) errors.

For example:

Yates A, Stark J, Klein N, Antia R, Callard R (2008) Correction: Understanding the Slow Depletion of Memory CD4+ T Cells in HIV Infection. PLoS Med 5(1): e11. https://doi.org/10.1371/journal.pmed.0050011

In another case, where there was a typo in a single equation that didn't propagate downstream, and the paper was older (>10 years from publication), the author confirmed the error but said (and I agreed) that publishing a correction probably wasn't worth the trouble.

Answer 4

Can we really trust the literature?

Let me point out here that "can a paper be trusted" and "all of the calculations in the paper are exactly correct" are not necessarily related. This is something that people get wrong about research, and in particular single papers, all the time. Academic papers are useful dialogue. They are not necessarily 100% accurate or unimpeachable.

Peer review is designed to catch methodological errors. It confirms that the authors followed sound scientific principles as appropriate to their field. Most peer review does not check for errors in calculation, and typically could not even if they wanted because the peer reviewer does not have access to all of the paper's data. However, peer review does confirm that a group of experts all read the paper and agreed that the methods and conclusion make sense.

A paper published in a good venue is not strong because it is guaranteed to be 100% accurate, it's strong because the expert community of reviewers thinks it is accurate. I would assume, though I can't cite any evidence, that academic papers at good venues have a much lower error rate than similar documents that do not go through a rigorous vetting process.

As to whether this error should be reported- it really depends. If this is a purely theoretical result, if it is empirically in doubt, if it substantially changes the result of the paper, then the error should really likely be reported. If the calculation is wrong but the intuition and the model are solid, and there is strong empirical evidence for the model, then the calculation error probably isn't that meaningful.

Here's a practical yardstick: if reporting the error substantially advances the scientific conversation, then it is a publishable result that should be published, just like any other publishable result. If reporting the error is just fixing errata and not substantially contributing to the body of knowledge, then it's not publishable and probably nobody cares.