29

Especially in the the more experimental subfields of computer science like systems, how often are results faked? If there is no verification process for any code used, do researchers sometimes fake results to save time?

  • 11
    Yes, how can you unintentionally fake data? – anon Apr 28 '15 at 12:27
  • 8
    Mistakes happen all the time. Also, many cases of fraud involve one person knowingly using false data and many others unwittingly accepting what that one person said. One retracted paper can have many authors, the majority of which were unintentionally part of the fraud. – eykanal Apr 28 '15 at 12:29
  • 4
    @eykanal: Then they weren't the ones who were faking the data. It seems pretty clear cut that the transitive verb "fake" implies intensionality. – Pete L. Clark Apr 28 '15 at 16:55
  • 17
    How do you expect anybody to be able to answer this? – Raphael Apr 28 '15 at 17:59
  • 5
    @RenéG If authors don't publish code to replicate their experiments (sometimes a difficult task), it can sometimes take a significant amount of work to re-implement their system, and often details that seem minor to the author and therefore don't end up in the paper can be important to the exact results they've seen. – Dougal Apr 29 '15 at 0:56
29

Nobody knows. Unless someone tries to reproduce the results and cannot, there's not ever even going to be a challenge to the results. Direct reproduction in CS and similar fields is generally done in papers that extend older results or propose a new method. To my knowledge, we do not see a lot of retractions based on these kinds of studies, so I'd say the rate of faking is low.

Retraction Watch's CS section has about 15 articles in it, but most of them appear to be about retractions for plagiarism not for faking results.

  • 10
    "Direct reproduction in CS and similar fields is generally done in papers that extend older results or propose a new method." It's unfortunate that it's very hard to get short papers of the form "yes, we were able to reproduce this result" published. – Joshua Taylor Apr 28 '15 at 17:28
  • 9
    Also, few papers publish executable code, and some not even pseudo code, along with their studies. Since we don't have a publication culture that appreciates effort of reproduction (cf @JoshuaTaylor) few results are checked. I suspect a high dark figure of published, peer-reviewed falsehoods. – Raphael Apr 28 '15 at 18:02
  • 16
    And the falsehoods aren't necessarily faked. Code has at least one bug in it. Suppose the bug in your unpublished code on which your paper relies happens to make it give incorrect but encouraging results, that you publish, and nobody ever checks. As a programmer, I'd say bugs that make your code appear to do what you wanted are the hardest to spot yourself. – Steve Jessop Apr 28 '15 at 21:11
  • 7
    @SteveJessop spot on. It must be very common for code to have bugs which give technically invalid results that nevertheless support the hypothesis. So the author is doing nothing wrong by publishing. However, one wonders how often a bug, even a significant one, might be discovered prior to publication and nevertheless conveniently ignored. – Keith Apr 29 '15 at 1:22
  • @SteveJessop Intentional or not, if the code is not there for people to check the authors don't act responsibly. – Raphael Apr 30 '15 at 17:28
18

Let me split my answer into two sub-areas: reputable venues and crap venues.

In reputable venues, it is just as possible for somebody to commit fraud (or any of the other deadly sins of science) in experimental computer science as in any other experimental science. It also appears to be quite rare, because there is usually pretty clear observability and a pretty clear relationship between theory and practice in computer science (unlike, say, certain subfields of biology) and so fraud would often be relatively easy to detect. More to the point, however, the risk/reward tradeoff is terrible: one detected incident will likely destroy a career.

In crap venues, there might well be constant fraud---but who cares? If they'll accept machine-generated papers, they might accept anything, and I'm probably not going to cite it or even look at it in any case.

  • Thanks for the link. While I knew about isolated cases (eg, a high rep user here published one as a proof of concept), this really surprised me. Are proceedings of the IEEE considered "crap" these days? They used to have a decent reputation, didn't they? – gnometorule Apr 28 '15 at 15:08
  • 5
    @gnometorule The IEEE in general is quite good, but they've been less careful than they should in exercising oversight as they have continued to expand. Thus, while the IEEE brand is a very good sign, you still have to exercise judgement just as you would for any other venue. – jakebeal Apr 28 '15 at 15:32
  • 5
    Hey, machines have just as much right to publish as humans do! Their papers may not be up to our standards yet, but they are trying their best to learn. :) – Thomas Apr 28 '15 at 16:08
  • 3
    "there is usually pretty clear observability and a pretty clear relationship between theory and practice in computer science" Though there's an unfortunate tendency to publish results that are based on computation/simulation/etc, without always making the code available. That means that reproducibility is actually much harder, because it's difficult to know whether a difference is a result of a bug, data, or the actual intended method. – Joshua Taylor Apr 28 '15 at 17:30
  • 5
    @jakebeal the IEEE brand is a very good sign — [citation needed] The quality of IEEE publications in computer science varies so widely that the label itself indicates nothing about the quality of the venue. (In this respect, they are no different than any other large commercial publisher.) – JeffE Apr 29 '15 at 2:19
11

There are quite a few interesting articles on the topic of fraud rates. I didn't see anything specific to computer science, but the general sense is that it's very difficult to determine.

It's very tough to actually detect fraud. Most cases (that I'm familiar with, at least) involve someone falsifying data in a highly active field and publishing earth-shattering results that turn out to be false. As was said in other answers, detecting the fraud requires attempting to reproduce the results, failing, and then determining that the problem is on the other end.

With that said, there aren't many good proxy measures of fraud. Retractions are a start, but they're far and few between. We can almost be certain that not all fraud is retracted. Surveys can be used but they're also notoriously inaccurate. There really aren't many other ways to measure that would actually provide a useful metric.

  • 3
    As most cases of exposed fraud happen due to researchers becoming to comfortable with falsifying data I think it's a very fair assessment to assume the large majority of fraud are not caught. – David Mulder Apr 28 '15 at 17:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy