29

Especially in the the more experimental subfields of computer science like systems, how often are results faked? If there is no verification process for any code used, do researchers sometimes fake results to save time?

Improve this question
9
  • 11
    Yes, how can you unintentionally fake data?
    – anon
    Apr 28, 2015 at 12:27
  • 8
    Mistakes happen all the time. Also, many cases of fraud involve one person knowingly using false data and many others unwittingly accepting what that one person said. One retracted paper can have many authors, the majority of which were unintentionally part of the fraud.
    – eykanal
    Apr 28, 2015 at 12:29
  • 4
    @eykanal: Then they weren't the ones who were faking the data. It seems pretty clear cut that the transitive verb "fake" implies intensionality.
    – Pete L. Clark
    Apr 28, 2015 at 16:55
  • 17
    How do you expect anybody to be able to answer this?
    – Raphael
    Apr 28, 2015 at 17:59
  • 5
    @RenéG If authors don't publish code to replicate their experiments (sometimes a difficult task), it can sometimes take a significant amount of work to re-implement their system, and often details that seem minor to the author and therefore don't end up in the paper can be important to the exact results they've seen.
    – Danica
    Apr 29, 2015 at 0:56

3 Answers 3

Reset to default
29

Nobody knows. Unless someone tries to reproduce the results and cannot, there's not ever even going to be a challenge to the results. Direct reproduction in CS and similar fields is generally done in papers that extend older results or propose a new method. To my knowledge, we do not see a lot of retractions based on these kinds of studies, so I'd say the rate of faking is low.

Retraction Watch's CS section has about 15 articles in it, but most of them appear to be about retractions for plagiarism not for faking results.

Improve this answer
7
  • 10
    "Direct reproduction in CS and similar fields is generally done in papers that extend older results or propose a new method." It's unfortunate that it's very hard to get short papers of the form "yes, we were able to reproduce this result" published.
    – Joshua Taylor
    Apr 28, 2015 at 17:28
  • 9
    Also, few papers publish executable code, and some not even pseudo code, along with their studies. Since we don't have a publication culture that appreciates effort of reproduction (cf @JoshuaTaylor) few results are checked. I suspect a high dark figure of published, peer-reviewed falsehoods.
    – Raphael
    Apr 28, 2015 at 18:02
  • 16
    And the falsehoods aren't necessarily faked. Code has at least one bug in it. Suppose the bug in your unpublished code on which your paper relies happens to make it give incorrect but encouraging results, that you publish, and nobody ever checks. As a programmer, I'd say bugs that make your code appear to do what you wanted are the hardest to spot yourself.
    – Steve Jessop
    Apr 28, 2015 at 21:11
  • 7
    @SteveJessop spot on. It must be very common for code to have bugs which give technically invalid results that nevertheless support the hypothesis. So the author is doing nothing wrong by publishing. However, one wonders how often a bug, even a significant one, might be discovered prior to publication and nevertheless conveniently ignored.
    – Keith
    Apr 29, 2015 at 1:22
  • @SteveJessop Intentional or not, if the code is not there for people to check the authors don't act responsibly.
    – Raphael
    Apr 30, 2015 at 17:28
18

Let me split my answer into two sub-areas: reputable venues and crap venues.

In reputable venues, it is just as possible for somebody to commit fraud (or any of the other deadly sins of science) in experimental computer science as in any other experimental science. It also appears to be quite rare, because there is usually pretty clear observability and a pretty clear relationship between theory and practice in computer science (unlike, say, certain subfields of biology) and so fraud would often be relatively easy to detect. More to the point, however, the risk/reward tradeoff is terrible: one detected incident will likely destroy a career.

In crap venues, there might well be constant fraud---but who cares? If they'll accept machine-generated papers, they might accept anything, and I'm probably not going to cite it or even look at it in any case.

Improve this answer
6
  • Thanks for the link. While I knew about isolated cases (eg, a high rep user here published one as a proof of concept), this really surprised me. Are proceedings of the IEEE considered "crap" these days? They used to have a decent reputation, didn't they?
    – gnometorule
    Apr 28, 2015 at 15:08
  • 5
    @gnometorule The IEEE in general is quite good, but they've been less careful than they should in exercising oversight as they have continued to expand. Thus, while the IEEE brand is a very good sign, you still have to exercise judgement just as you would for any other venue.
    – jakebeal
    Apr 28, 2015 at 15:32
  • 5
    Hey, machines have just as much right to publish as humans do! Their papers may not be up to our standards yet, but they are trying their best to learn. :)
    – Thomas
    Apr 28, 2015 at 16:08
  • 3
    "there is usually pretty clear observability and a pretty clear relationship between theory and practice in computer science" Though there's an unfortunate tendency to publish results that are based on computation/simulation/etc, without always making the code available. That means that reproducibility is actually much harder, because it's difficult to know whether a difference is a result of a bug, data, or the actual intended method.
    – Joshua Taylor
    Apr 28, 2015 at 17:30
  • 5
    @jakebeal the IEEE brand is a very good sign — [citation needed] The quality of IEEE publications in computer science varies so widely that the label itself indicates nothing about the quality of the venue. (In this respect, they are no different than any other large commercial publisher.)
    – JeffE
    Apr 29, 2015 at 2:19
11

There are quite a few interesting articles on the topic of fraud rates. I didn't see anything specific to computer science, but the general sense is that it's very difficult to determine.

It's very tough to actually detect fraud. Most cases (that I'm familiar with, at least) involve someone falsifying data in a highly active field and publishing earth-shattering results that turn out to be false. As was said in other answers, detecting the fraud requires attempting to reproduce the results, failing, and then determining that the problem is on the other end.

With that said, there aren't many good proxy measures of fraud. Retractions are a start, but they're far and few between. We can almost be certain that not all fraud is retracted. Surveys can be used but they're also notoriously inaccurate. There really aren't many other ways to measure that would actually provide a useful metric.

Improve this answer
1
  • 3
    As most cases of exposed fraud happen due to researchers becoming to comfortable with falsifying data I think it's a very fair assessment to assume the large majority of fraud are not caught.
    – David Mulder
    Apr 28, 2015 at 17:17

You must log in to answer this question.