23

I am preparing a paper in the field of Computer Science.

In order to report test results, we usually run a number of tests and report the average of those tests.

For each test, we generate random data.

Because of the randomness, at some points, the results may come out not as expected.

For instance, a graph may be like: enter image description here

Usually, one should explain why on points 8, 11 and 12 there is a decrease on the plot. Probably, it is because of that randomness.

Not hand-crafting all the graph, but just manipulating a few points makes the graph acceptable: enter image description here

Since three weeks or so, I work my ass off and try to figure out why my resulting graph looks like the first one. Sometimes I feel like yielding to temptation and just modify the raw data before I go crazy.

I believe, at this point the title became misleading, so let me make it clear:

I am not seeking an advice on data manipulation. I will not manipulate my data. However, I ask to myself "how the hell this can be detected?"

And now, I don't only ask to myself, but to whole community. How is this detected? For editors, referees out there, have you ever detected something like this?

  • 8
    Many of the posts in the RetractionWatch image manipulation tag describe how the manipulation was caught. There isn't much CS representation over there, though. – ff524 Dec 4 '14 at 0:24
  • 2
    Related (possible duplicate?): What should raise red flags to detect fabricated data – ff524 Dec 4 '14 at 4:16
  • 10
    If using random data can cause sizeable movement in your data points, simply be honest and include error bars on them. Yes, that does mean you will have to work extra to figure out how big those error bars should be and what they mean exactly, but it's exactly that work that will enable you to tell the reviewers that the downward dips are artefacts due to your statistics. – E.P. Dec 4 '14 at 11:34
  • 14
    By the way, I'd advise you to let go of the idea that a plot is unacceptable or in any way undesirable simply because it's not "smooth enough". Scientists know that results from a random procedure will be randomly distributed, they will not count that distribution against you. – David Z Dec 4 '14 at 13:27
  • 2
    Since you generate random data, it would be good to repeat tests a couple of times and for each data point, i.e. your 1 until 14, report a 95% confidence interval. This way it will be clear to readers that the actual value probably lies in each reported range and the reader will understand that the obtained points cannot lie on a perfect line. – Ritz Dec 5 '14 at 9:19
24

The image manipulations reported on Retraction Watch are most of the time naive collages of gel photographs or spectrograms. They get caught, among other things, because repeating patterns in the noise appear on closer inspection, or linear disruption of the noise are visible, see this.

For 1D data, the case you mention, there is the Benford's law and other statistical tests that can indicate potential manipulation of data. It usually relies on human beings preferring certain digits over others, even unconsciously, thus generating data that has a non-random variability.

Also, many journals ask for graphs to be submitted in vector format, which means you are actually sending the data points, and not just a rendered figure. Things like editing out a few data points to smooth a curve will be apparent.

Now, to the best of my knowledge publishers and, even less so, reviewers don't systematically screen for these things, they only do so if they have suspicions, because the scientific publishing process is based on good faith. But if the paper gets any sort of attention it will get caught by post publication review.

Don't fabricate/manipulate data. It's adding unwanted noise to an already noisy signal, it's dishonest towards your coworkers, the people who fund you, the publisher and the readership, and it will ruin your career.

  • 10
    In addition to Benford's law, you can use goodness-of-fit tests to see if the data fits the model "too well." There are various papers on detecting fabricated data, e.g., this one. – Kimball Dec 4 '14 at 2:01
  • 3
    This page describes in great detail the techniques and software used by the US Office of Research Integrity to detect the kinds of falsification mentioned in this answer. – ff524 Dec 4 '14 at 4:22
14

Cape Code pointed out that in fields that involve use of gel photographs or spectrograms, sloppy image manipulation can be detected by experienced readers.

In other fields, data can be flagged as possibly fraudulent for being "too perfect." For example, here is the abstract of a report that led to the investigation of a social psychology researcher:

Here we analyze results from three recent papers (2009, 2011, 2012) by Dr. Jens Förster from the Psychology Department of the University of Amsterdam. These papers report 40 experiments involving a total of 2284 participants (2242 of which were undergraduates). We apply an F test based on descriptive statistics to test for linearity of means across three levels of the experimental design. Results show that in the vast majority of the 42 independent samples so analyzed, means are unusually close to a linear trend. Combined left-tailed probabilities are 0.000000008, 0.0000004, and 0.000000006, for the three papers, respectively. The combined left-tailed p-value of the entire set is p= 1.96 * 10-21, which corresponds to finding such consistent results (or more consistent results) in one out of 508 trillion (508,000,000,000,000,000,000). Such a level of linearity is extremely unlikely to have arisen from standard sampling. We also found overly consistent results across independent replications in two of the papers. As a control group, we analyze the linearity of results in 10 papers by other authors in the same area. These papers differ strongly from those by Dr. Förster in terms of linearity of effects and the effect sizes. We also note that none of the 2284 participants showed any missing data, dropped out during data collection, or expressed awareness of the deceit used in the experiment, which is atypical for psychological experiments.

This report is obviously the result of some non-trivial effort. But some of the symptoms described (exceptionally good fit, no experiment participants dropping out, atypically large effect sizes) can raise alarms for any experienced, diligent reviewer, possibly leading to a more formal investigation.

  • In addition to these, missing IRB approvals indicates one of two serious ethical breaches has occurred. – Fomite Dec 4 '14 at 3:56
10

First of all, don't do it.

You probably wouldn't be detected, because peer review isn't generally hunting for subtle data manipulation. Methods like those the answer by CapeCode could be applied, but even then a small number of data points like you are showing would not likely produce a terribly conclusive indication of dishonesty. But it will be in the literature forever, and you never know...

But really, that doesn't matter. Whether or not you get detected, you will certainly still know you that you lied. You'll be voluntarily throwing out the one thing that nobody can take from you: your integrity. Will it stop there, or will you do it again, the next time something's not quite perfect? How much of your work will be tainted? Pretty much all of us researchers struggle with impostor syndrome, but if you go down this path, you'll know it's true. Do you really want to live that way?

Not only that, but you will have lied and compromised yourself over something really stupid, just to make a graph a little bit prettier. If you have real results, they will stand, even with noise. If the noise is big enough to actually be a problem, then that's not a problem, that's an opportunity. As the quote attributed to Asimov goes:

The most exciting phrase to hear in science, the one that heralds new discoveries, is not “Eureka” but “That’s funny...”

A lot of important emergent phenomena in computer science get discovered that way as well. If you lie, not only are you compromising your integrity and risking total damnation if it ever gets discovered, but you are also cutting off the possibility that you might stumble over something more important than what you were doing at first.

In short: don't do it.

  • 4
    Per the question: "I will not manipulate my data." While this post is good advice generally speaking, it's (1) I think, a little insulting to the OP to ignore that statement, and (2) doesn't add anything to the existing answer about the actual subject of the question (detecting data manipulation) – ff524 Dec 4 '14 at 3:19
  • 2
    @ff524 I read both that and "Sometimes I feel like yielding to temptation." Maybe I'm reacting too much to the second, but I feel it's ambivalent as presented, and I'm happy to let the community judge how useful my response is. – jakebeal Dec 4 '14 at 4:43
9

Why not run the experiment enough times so that you can produce your plot with error bars on the points? This will make it possible for the reader to understand how much random variation there is in the measurements.

  • 5
    While this is good advice, it's not an answer to the question about detecting data manipulation - it should be a comment on the question instead. – ff524 Dec 4 '14 at 3:13
  • 1
    Brian, it's not always possible to do that either - some measurements are costly or time-consuming. – smci Dec 5 '14 at 1:22
3

Others have provided useful input but I am not sure they completely addressed the “How can editors and reviewers detect data manipulation?” question.

The simple answer is that mostly, they can't and they don't, certainly not in fields where researchers don't routinely share code, raw data, photographs and the like but only statistical tests or basic plots. If you are really sloppy, you might end up with incoherent numbers that could not possibly have been produced by the analysis you claim to have done (I have seen things like that) but more subtle manipulation is not so easy to detect.

There are a few fascinating techniques to detect bogus data (including but not limited to Benford's law) but very few people actually have the expertise required and reviewers do not routinely check for that. In most cases, such an analysis can give you a strong presumption but no solid proof. Some famous data sets have been thoroughly analyzed without reaching a consensus (e.g. Cyril Burt's work on intelligence and heredity).

If you look at some of the high profile cases of fraud exposed in recent years (Jens Förster but also Diederik Stapel or Dirk Smeesters), they were mostly found out after many many fraudulent publications and not always because there was anything suspicious about these publications. The more “greedy” the fraudster is, the clearer the pattern becomes and some people might have had private misgivings at some stage but the fraud is only exposed later, usually after someone blew the whistle and not because a reviewer noticed it.

You can look at this as a glass half full (fraud is eventually detected) or half empty (How could it go on for so long? How many others are out there?) but the fact is that it's only in the aggregate that the results look suspicious, not at the level of a single graph or article.

Not that I advocate doing that, of course. Ethically, it's clearly wrong and the cases I just mentioned show that you can get found out in other ways and face very serious consequences. But reviewers and editors usually can't detect fraud directly, that's not how the systems works.

  • Also, the technieques to detect bogus data cannot do miracles. It should not be too difficult to produce bogus data that cannot easily be differentiated from real data, even though that happens too. In the end, the only reliable way is to try to reproduce the whole experiment or parts of it and for that you would need all the relevant information (algorithms, implementations, protocols, substances, ...) to be available next to the publication. – Trilarion Apr 6 '17 at 11:37
0

At the point at which you only have the figure, or the underlying processed data, you cannot detect "well crafted" manipulation. One aspect of reproducible research, which is becoming more popular, requires that others be able to reproduce the data. This means making code available, describing hardware in sufficient detail, and also proving things like seeds and states of random number generators. This allows reviewers to recreate your data and then test how sensitive they are to slight perturbations.

  • Care to explain the downvote? – StrongBad Dec 5 '14 at 13:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.