How accurate are published papers?

Question

I'm an undergrad trying to publish findings on the accuracy of consumer-grade motion sensors for IEEE. I've searched for existing literature on this topic, and I've found some interesting articles. However, I'm not sure how accurate these papers are.

Once IEEE publishes a paper in a journal or conference, has IEEE deemed the paper's findings as accurate?

That is, has a paper passed science's often-vaunted "peer-review" once that paper has been published?

I realize this question applies for my research and any other research published by a respected organization like IEEE or Nature.

Answer 1

No. The publisher does not and cannot guarantee the correctness of the papers. Also the peer-review system is not perfect.

That being said, even though a single paper might have some chance of being wrong, as studies are replicated, and follow up studies verify and extend the conclusions, the scientific community can build up stronger claims.

Answer 2

One paper specifically looked at Why Most Published Research Findings Are False and stimulated a lot of discussion around the topic.

EDIT to provide more context, as requested in comments.
The paper (published in PLOS Medicine, which defines its scope) uses statistical methods to examine the probabilities that specific study types in biomedical research will yield significant results that end up being published.

Some quotes:

the high rate of nonreplication (lack of confirmation) of research discoveries is a consequence of the convenient, yet ill-founded strategy of claiming conclusive research findings solely on the basis of a single study assessed by formal statistical significance, typically for a p-value less than 0.05.

---

most research questions are addressed by many teams, and it is misleading to emphasize the statistically significant findings of any single team. What matters is the totality of the evidence. Diminishing bias through enhanced research standards and curtailing of prejudices may also help. However, this may require a change in scientific mentality that might be difficult to achieve.

---

Despite a large statistical literature for multiple testing corrections [37], usually it is impossible to decipher how much data dredging by the reporting authors or other research teams has preceded a reported research finding. Even if determining this were feasible, this would not inform us about the pre-study odds.

While some details of the methodology have been disputed, the gist of the article – that replication of studies is crucial to establish facts, and that the current publication system is biased against replicable research in many ways – has been widely acknowledged well beyond biomedicine, and the paper has triggered a wave of studies examining this range of systemic problems that the paper highlighted.

Answer 3

Publication in a peer reviewed journal is only the start of the road to the acceptance of an idea by the research community, not the last. Peer review should really only be regarded as a basic sanity check of the paper and not relied upon. Once the findings of a paper have been replicated, or used as the basis for further research, then we can be more confident that the paper is correct, but it is still the readers responsibility to make sure they judge the paper for themselves. If a paper has been cited a large number of times (without the citations being refutations! ;o), then that suggest the work is probably sound. Uncritically trusting a paper because it has been peer-reviewed is essentially an example of the "argument from authority" fallacy (in this case the anonymous reviewers being the authority). While peer-reviewed papers are more likely to be correct than, say blogs, there can be no guarantee of this, and it is best to maintain a skeptical attitude.

Answer 4

The accuracy of published papers varies dramatically and can not be assumed. Specifically in the case of IEEE, it's a huge organization with lots of conferences and journals. Some of them tend to be more strict about what they accept than others. Some of the papers I've read from IEEE made me wonder whether they went through any peer review process at all, as the language wasn't even comprehensible. Even in the most respected journals and conferences, though, results should not be assumed accurate. You should use your own judgment to evaluate the logic and arguments presented (i.e. to ensure that their conclusions really do follow from their premises/experimental results and that their experimental designs seem reasonable) and, where feasible, repeat the experiments to verify their results.

At any rate, the peer-review process that papers go through for publication is not a verification of the results, even in the most respected journals. The peer-review process consists of other authors looking over the paper and pointing out any perceived weaknesses in the arguments presented, noticeably incorrect results (i.e. math errors and the like,) and/or formatting and language errors. In some cases (especially for more general conferences,) the reviewer isn't even in the same research field as the paper (e.g. an RF engineer might be reviewing a paper on power systems or an AI researcher might be reviewing a paper on computer networks.) IEEE publishing a paper is basically saying "After a cursory review, one or more members of IEEE think that this paper doesn't appear to be complete trash and likely has some academic value to someone," not "IEEE asserts on the basis of its reputation that this paper's results are correct."

The process you seem to have in mind - other researchers repeating the experiments to verify the results - is a completely distinct process from the peer-review process required for publication. This process usually doesn't start until after the paper is published. This is generally true for all scientific publication, not just IEEE. Mathematics is something of an exception in that they attempt to present actual proofs rather than scientific results in their papers. That is also often the case in the algorithms and logic side of Computer Science. Even in those fields, though, you should check the proofs yourself, not assume them to be accurate just because they got published.

Especially where it is not feasible to reproduce a paper's results yourself, it is perfectly acceptable to cite that paper's results in your paper as a comparison. You just need to make sure that you cite it as something along the lines of "<authors of other paper> found that their system achieved X performance level, while our experiments show that our system achieves Y performance level." That way, it's clear that you're not claiming to have verified their results, but that you're merely comparing what you saw from your system to what they claim to have seen from theirs. Of course, if it's actually feasible for you to test the other system yourself and present your own experimental results for it, that's better.

Answer 5

In addition to other good answers/points, there is also the phenomenon that a result might seem approximately correct, and not looked at toooo critically because it didn't seem to really matter much... but met basic professional competence criteria, etc. If it happens to be the basis of other innocent/boring work, things can pile up on assumptions of correctness that have not been "stressed" at all. Then, if some more-serious/interesting development arises that depends on all that, people will reconsider everything in a much more serious way. Obviously! But, and until, that latter degree of "interest" has occurred, I'd worry that the chances are "higher than we'd like" that there're problems in a given result. Possibly reparable problems, especially in "routine" stories, but repair might often require a more expert appreciation of the situation.

Summary: if a result has been "stressed", not merely cited, one can have some confidence. But even if often-cited, but only by uninteresting/inconsequential papers, who knows?

And, as people have noted, many journals (in mathematics) specifically tell the referee that it's not their responsibility to check for correctness, but more/only for "suitability". (A status issue, among others, ...)

Answer 6

Many papers in medicine, biology, psychology, social sciences, etc rely on statistical confidence levels, and so come with an explicit statement that they cannot be seen to be 100% accurate.

A typical confidence level is 95%, which means unfortunately that you can expect roughly 1 in 20 conclusions to be incorrect. So these are fields where duplicated or reinforcing results are important.

Answer 7

At the very least, we generally make assumptions and are - unconsciously - looking for them. One particular manifestation is P-hacking. See Dr Derek Muller explanation on this:

Most Published Research Wrong? https://www.youtube.com/watch?v=42QuXLucH3Q

Also, incorrect use of Variance Reduction Techniques, random number generators and meshing can contribute towards inaccuracies.

---

It is natural that reproducibility becomes a concern.

When graduate student Alyssa Ward took a science-policy internship, she expected to learn about policy — not to unearth gaps in her biomedical training.

She was compiling a bibliography about the reproducibility of experiments, and one of the papers, a meta-analysis, found that scientists routinely fail to explain how they choose the number of samples to use in a study. “My surprise was not about the omission — it was because I had no clue how, or when, to calculate sample size,” Ward says. Nor had she ever been taught about major categories of experimental design, or the limitations of P values. (Although they can help to judge the strength of scientific evidence, P values do not — as many think — estimate the likelihood that a hypothesis is true.)

Reproducibility: Seek out stronger science

A P value of 0.05 does not mean that there is a 95% chance that a given hypothesis is correct. Instead, it signifies that if the null hypothesis is true, and all other assumptions made are valid, there is a 5% chance of obtaining a result at least as extreme as the one observed. And a P value cannot indicate the importance of a finding; for instance, a drug can have a statistically significant effect on patients’ blood glucose levels without having a therapeutic effect.

Statisticians issue warning over misuse of P values

Answer 8

Let's put this way:

• They are not certainly-accruate enough to be valid and accurate for you to base your paradigms on without a lot of additional results and publications - but
• They are certainly-accurate enough for you to be able to entertain the opposite claim without presenting iron-clad arguments and/or experimental results to counter them.