29

I'm an undergrad trying to publish findings on the accuracy of consumer-grade motion sensors for IEEE. I've searched for existing literature on this topic, and I've found some interesting articles. However, I'm not sure how accurate these papers are.

Once IEEE publishes a paper in a journal or conference, has IEEE deemed the paper's findings as accurate?

That is, has a paper passed science's often-vaunted "peer-review" once that paper has been published?

I realize this question applies for my research and any other research published by a respected organization like IEEE or Nature.

  • 29
    It has surely passed a peer review process, but I don't think it means what you think it does. "Peer reviewed" means just that. It's not evidence of correctness or a guarantee or anything remotely like that. Even mathematicians (many of whom think referees should check correctness very carefully, which may not be the case in other fields) don't blindly believe results just because they're published in a reputable peer reviewed journal. – Yuichiro Fujiwara Mar 17 '15 at 19:26
  • 4
    @Yuichiro: Not a guarantee, certainly, but to say that publication in a reputable general is "not evidence of correctness" seems a bit much. Also, mathematicians sometimes blindly believe results when they are published in sufficiently reputable journals. When and how much they do is a quite complicated issue, I think... – Pete L. Clark Mar 18 '15 at 18:20
  • 1
    @PeteL.Clark Good point. By "not evidence," I meant that it's not conclusive evidence or the smoking gun. It can indirectly suggest a result is correct. So I agree it could be called evidence of some sort if you say other somewhat reliable indicators like the authors' solid track records are also "evidence." The latter point you made touches too complicated an issue for this comment section, I think. But, let's say, what would your answer be when asked if you blindly believe a paper just because it's published in Annals? If it's not simple yes, I think we're not disagreeing. – Yuichiro Fujiwara Mar 18 '15 at 19:28
  • 1
    Professors don't get tenure by repeating other group's experiments to check for accuracy. – user137 Mar 18 '15 at 23:42
41

No. The publisher does not and cannot guarantee the correctness of the papers. Also the peer-review system is not perfect.

That being said, even though a single paper might have some chance of being wrong, as studies are replicated, and follow up studies verify and extend the conclusions, the scientific community can build up stronger claims.

  • 7
    ...and should not... – JeffE Mar 17 '15 at 21:27
  • 4
    Depending on your field, average quality may be ... questionable (especially on conferences) and follow-up verification next to non-existing. In particular, in very active areas (i.e. such with lots of publications) it's generally understood (from what I hear) that "peer-review" means little more than "was read twice", if that. – Raphael Mar 18 '15 at 13:23
  • In addition, scientific papers (try to) push the boundaries of knowledge. There may be pitfalls/influencing factors that neither authors nor reviewers are aware of because they are still unknown. – cbeleites supports Monica Oct 20 '16 at 23:15
16

One paper specifically looked at Why Most Published Research Findings Are False and stimulated a lot of discussion around the topic.

EDIT to provide more context, as requested in comments.
The paper (published in PLOS Medicine, which defines its scope) uses statistical methods to examine the probabilities that specific study types in biomedical research will yield significant results that end up being published.

Some quotes:

the high rate of nonreplication (lack of confirmation) of research discoveries is a consequence of the convenient, yet ill-founded strategy of claiming conclusive research findings solely on the basis of a single study assessed by formal statistical significance, typically for a p-value less than 0.05.


most research questions are addressed by many teams, and it is misleading to emphasize the statistically significant findings of any single team. What matters is the totality of the evidence. Diminishing bias through enhanced research standards and curtailing of prejudices may also help. However, this may require a change in scientific mentality that might be difficult to achieve.

Despite a large statistical literature for multiple testing corrections [37], usually it is impossible to decipher how much data dredging by the reporting authors or other research teams has preceded a reported research finding. Even if determining this were feasible, this would not inform us about the pre-study odds.

While some details of the methodology have been disputed, the gist of the article – that replication of studies is crucial to establish facts, and that the current publication system is biased against replicable research in many ways – has been widely acknowledged well beyond biomedicine, and the paper has triggered a wave of studies examining this range of systemic problems that the paper highlighted.

  • 7
    Not to say these aren't great references, but could you describe them a bit in your answer? Some people may be accessing this site through a channel that, for one reason or another, blocks the websites you linked from, and though they are recent and active links, you should safeguard this answer from suffering from Link Rot. – Zibbobz Mar 18 '15 at 14:52
  • 1
    @gnometorule I believe the answer, I'm just saying it would be improved with supporting quotes. – Zibbobz Mar 18 '15 at 22:58
  • @gnometorule In general on SE sites, answers should stand on their own; links are supplemental. – Kyle Strand Mar 18 '15 at 23:26
  • @Zibbobz I edited my answer to provide some details about the paper. – Daniel Mietchen Mar 19 '15 at 0:19
9

Publication in a peer reviewed journal is only the start of the road to the acceptance of an idea by the research community, not the last. Peer review should really only be regarded as a basic sanity check of the paper and not relied upon. Once the findings of a paper have been replicated, or used as the basis for further research, then we can be more confident that the paper is correct, but it is still the readers responsibility to make sure they judge the paper for themselves. If a paper has been cited a large number of times (without the citations being refutations! ;o), then that suggest the work is probably sound. Uncritically trusting a paper because it has been peer-reviewed is essentially an example of the "argument from authority" fallacy (in this case the anonymous reviewers being the authority). While peer-reviewed papers are more likely to be correct than, say blogs, there can be no guarantee of this, and it is best to maintain a skeptical attitude.

  • 3
    Citation count is not a great metric for the reliability of a paper. Papers are often cited uncritically -- often just because the author needs to flesh out a narrative and make their own pet theory seem plausible. I've seen this in my own field -- a paper was published in Science that should not have gotten past peer review. Yet this paper has received a large number of citations -- mainly from people in a different field who just liked the conclusion. There are only a few citations from within the field, and these were rejections of the paper. The large citation count just demonstrates a fad – adam.r Mar 22 '15 at 15:36
  • @adam.r no metric is perfect, however citation count has the advantage of being available. While some papers are cited uncritically, the evidence that they have at least been cited suggests that they have been subject to more scrutiny than papers that haven't. To say that a large citation count just demonstrates a fad is pretty insulting towards the majority of well cited papers that are well cited because they are worth citing. – Dikran Marsupial Mar 23 '15 at 8:01
  • 1
    I did not say that citations are typically due to fads, just that some papers gain many citations due to fads. (and I did vote-up your answer). My point was to provide guidance to the interpreting that number. For instance, high-profile journals like Science are more likely to get fad citations than workhorse journals. Fad citations also show up soon after publication (with no time for replication), but solid papers get citations for many years. No statistic is perfect, but the only way to get better stats is to understand the limits of existing stats. – adam.r Mar 23 '15 at 15:58
  • 1
    @adam.r fair enough, sorry for the misunderstanding. It is also true that there are a fair few papers that get frequently cited for a number of years with algorithms that don't really work very well in practice. I did say that being highly cited only suggests the paper is "probably sound". At the end of the day the only way to find out is often to try it for yourself and see if it works and we need to retain skepticism. – Dikran Marsupial Mar 23 '15 at 16:14
  • 1
    Marupial: No problem. My original comment ("not a great metric") was open for interpretation. – adam.r Mar 24 '15 at 12:04
2

The accuracy of published papers varies dramatically and can not be assumed. Specifically in the case of IEEE, it's a huge organization with lots of conferences and journals. Some of them tend to be more strict about what they accept than others. Some of the papers I've read from IEEE made me wonder whether they went through any peer review process at all, as the language wasn't even comprehensible. Even in the most respected journals and conferences, though, results should not be assumed accurate. You should use your own judgment to evaluate the logic and arguments presented (i.e. to ensure that their conclusions really do follow from their premises/experimental results and that their experimental designs seem reasonable) and, where feasible, repeat the experiments to verify their results.

At any rate, the peer-review process that papers go through for publication is not a verification of the results, even in the most respected journals. The peer-review process consists of other authors looking over the paper and pointing out any perceived weaknesses in the arguments presented, noticeably incorrect results (i.e. math errors and the like,) and/or formatting and language errors. In some cases (especially for more general conferences,) the reviewer isn't even in the same research field as the paper (e.g. an RF engineer might be reviewing a paper on power systems or an AI researcher might be reviewing a paper on computer networks.) IEEE publishing a paper is basically saying "After a cursory review, one or more members of IEEE think that this paper doesn't appear to be complete trash and likely has some academic value to someone," not "IEEE asserts on the basis of its reputation that this paper's results are correct."

The process you seem to have in mind - other researchers repeating the experiments to verify the results - is a completely distinct process from the peer-review process required for publication. This process usually doesn't start until after the paper is published. This is generally true for all scientific publication, not just IEEE. Mathematics is something of an exception in that they attempt to present actual proofs rather than scientific results in their papers. That is also often the case in the algorithms and logic side of Computer Science. Even in those fields, though, you should check the proofs yourself, not assume them to be accurate just because they got published.

Especially where it is not feasible to reproduce a paper's results yourself, it is perfectly acceptable to cite that paper's results in your paper as a comparison. You just need to make sure that you cite it as something along the lines of "<authors of other paper> found that their system achieved X performance level, while our experiments show that our system achieves Y performance level." That way, it's clear that you're not claiming to have verified their results, but that you're merely comparing what you saw from your system to what they claim to have seen from theirs. Of course, if it's actually feasible for you to test the other system yourself and present your own experimental results for it, that's better.

  • It may be helpful to note that the process of verification may be best represented in review papers, where the author has taken pains to put each publication in the context and illustrate which findings were the basis for subsequent studies, and which theories held up to multiple independent tests. Still, some review papers are junk and amount to little more than a scientist waving his own flag as a fundraising exercise. – adam.r Mar 22 '15 at 15:43
  • @adam.r Exactly how the verification process works varies somewhat by field, though, which is part of why I omitted that discussion (aside from it probably deserving its own question.) IEEE (as the name implies - The Institute for Electrical and Electronics Engineers) is more focused on engineering than on pure science, so the process tends to be somewhat different than in the physical sciences. – reirab Mar 22 '15 at 19:20
2

In addition to other good answers/points, there is also the phenomenon that a result might seem approximately correct, and not looked at toooo critically because it didn't seem to really matter much... but met basic professional competence criteria, etc. If it happens to be the basis of other innocent/boring work, things can pile up on assumptions of correctness that have not been "stressed" at all. Then, if some more-serious/interesting development arises that depends on all that, people will reconsider everything in a much more serious way. Obviously! But, and until, that latter degree of "interest" has occurred, I'd worry that the chances are "higher than we'd like" that there're problems in a given result. Possibly reparable problems, especially in "routine" stories, but repair might often require a more expert appreciation of the situation.

Summary: if a result has been "stressed", not merely cited, one can have some confidence. But even if often-cited, but only by uninteresting/inconsequential papers, who knows?

And, as people have noted, many journals (in mathematics) specifically tell the referee that it's not their responsibility to check for correctness, but more/only for "suitability". (A status issue, among others, ...)

1

Many papers in medicine, biology, psychology, social sciences, etc rely on statistical confidence levels, and so come with an explicit statement that they cannot be seen to be 100% accurate.

A typical confidence level is 95%, which means unfortunately that you can expect roughly 1 in 20 conclusions to be incorrect. So these are fields where duplicated or reinforcing results are important.

  • 4
    That 1 in 20 is assuming that they did the statistical analysis correctly... – Jaap Eldering Mar 18 '15 at 10:27
  • 3
    And that no publication bias towards positive results has been applied. – Jack Aidley Mar 18 '15 at 10:33
  • The perverse thing is that - even though they are so important - confirmatory results are very hard to get published in top tier journals, since the results are then easily discarded as not novel. – Marc Claesen Mar 18 '15 at 10:46
  • More generally, pretty much all science does this (using confidence intervals rather than proofs, that is.) That's how science works. Science doesn't give you proven results; it gives you statistical evidence to support your hypothesis. In non-science fields, though (like math and the math side of CS,) papers do often contain actual proofs. – reirab Mar 18 '15 at 15:47
0

At the very least, we generally make assumptions and are - unconsciously - looking for them. One particular manifestation is P-hacking. See Dr Derek Muller explanation on this:

Most Published Research Wrong? https://www.youtube.com/watch?v=42QuXLucH3Q

Also, incorrect use of Variance Reduction Techniques, random number generators and meshing can contribute towards inaccuracies.


It is natural that reproducibility becomes a concern.

When graduate student Alyssa Ward took a science-policy internship, she expected to learn about policy — not to unearth gaps in her biomedical training.

She was compiling a bibliography about the reproducibility of experiments, and one of the papers, a meta-analysis, found that scientists routinely fail to explain how they choose the number of samples to use in a study. “My surprise was not about the omission — it was because I had no clue how, or when, to calculate sample size,” Ward says. Nor had she ever been taught about major categories of experimental design, or the limitations of P values. (Although they can help to judge the strength of scientific evidence, P values do not — as many think — estimate the likelihood that a hypothesis is true.)

Reproducibility: Seek out stronger science

A P value of 0.05 does not mean that there is a 95% chance that a given hypothesis is correct. Instead, it signifies that if the null hypothesis is true, and all other assumptions made are valid, there is a 5% chance of obtaining a result at least as extreme as the one observed. And a P value cannot indicate the importance of a finding; for instance, a drug can have a statistically significant effect on patients’ blood glucose levels without having a therapeutic effect.

Statisticians issue warning over misuse of P values

0

Let's put this way:

  • They are not certainly-accruate enough to be valid and accurate for you to base your paradigms on without a lot of additional results and publications - but
  • They are certainly-accurate enough for you to be able to entertain the opposite claim without presenting iron-clad arguments and/or experimental results to counter them.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.