How can editors and reviewers detect data manipulation?

Question

I am preparing a paper in the field of Computer Science.

In order to report test results, we usually run a number of tests and report the average of those tests.

For each test, we generate random data.

Because of the randomness, at some points, the results may come out not as expected.

For instance, a graph may be like:

Usually, one should explain why on points 8, 11 and 12 there is a decrease on the plot. Probably, it is because of that randomness.

Not hand-crafting all the graph, but just manipulating a few points makes the graph acceptable:

Since three weeks or so, I work my ass off and try to figure out why my resulting graph looks like the first one. Sometimes I feel like yielding to temptation and just modify the raw data before I go crazy.

I believe, at this point the title became misleading, so let me make it clear:

I am not seeking an advice on data manipulation. I will not manipulate my data. However, I ask to myself "how the hell this can be detected?"

And now, I don't only ask to myself, but to whole community. How is this detected? For editors, referees out there, have you ever detected something like this?

Answer 1

The image manipulations reported on Retraction Watch are most of the time naive collages of gel photographs or spectrograms. They get caught, among other things, because repeating patterns in the noise appear on closer inspection, or linear disruption of the noise are visible, see this.

For 1D data, the case you mention, there is the Benford's law and other statistical tests that can indicate potential manipulation of data. It usually relies on human beings preferring certain digits over others, even unconsciously, thus generating data that has a non-random variability.

Also, many journals ask for graphs to be submitted in vector format, which means you are actually sending the data points, and not just a rendered figure. Things like editing out a few data points to smooth a curve will be apparent.

Now, to the best of my knowledge publishers and, even less so, reviewers don't systematically screen for these things, they only do so if they have suspicions, because the scientific publishing process is based on good faith. But if the paper gets any sort of attention it will get caught by post publication review.

Don't fabricate/manipulate data. It's adding unwanted noise to an already noisy signal, it's dishonest towards your coworkers, the people who fund you, the publisher and the readership, and it will ruin your career.

Answer 2

Cape Code pointed out that in fields that involve use of gel photographs or spectrograms, sloppy image manipulation can be detected by experienced readers.

In other fields, data can be flagged as possibly fraudulent for being "too perfect." For example, here is the abstract of a report that led to the investigation of a social psychology researcher:

Here we analyze results from three recent papers (2009, 2011, 2012) by Dr. Jens Förster from the Psychology Department of the University of Amsterdam. These papers report 40 experiments involving a total of 2284 participants (2242 of which were undergraduates). We apply an F test based on descriptive statistics to test for linearity of means across three levels of the experimental design. Results show that in the vast majority of the 42 independent samples so analyzed, means are unusually close to a linear trend. Combined left-tailed probabilities are 0.000000008, 0.0000004, and 0.000000006, for the three papers, respectively. The combined left-tailed p-value of the entire set is p= 1.96 * 10-21, which corresponds to finding such consistent results (or more consistent results) in one out of 508 trillion (508,000,000,000,000,000,000). Such a level of linearity is extremely unlikely to have arisen from standard sampling. We also found overly consistent results across independent replications in two of the papers. As a control group, we analyze the linearity of results in 10 papers by other authors in the same area. These papers differ strongly from those by Dr. Förster in terms of linearity of effects and the effect sizes. We also note that none of the 2284 participants showed any missing data, dropped out during data collection, or expressed awareness of the deceit used in the experiment, which is atypical for psychological experiments.

This report is obviously the result of some non-trivial effort. But some of the symptoms described (exceptionally good fit, no experiment participants dropping out, atypically large effect sizes) can raise alarms for any experienced, diligent reviewer, possibly leading to a more formal investigation.

Answer 3

First of all, don't do it.

You probably wouldn't be detected, because peer review isn't generally hunting for subtle data manipulation. Methods like those the answer by CapeCode could be applied, but even then a small number of data points like you are showing would not likely produce a terribly conclusive indication of dishonesty. But it will be in the literature forever, and you never know...

But really, that doesn't matter. Whether or not you get detected, you will certainly still know you that you lied. You'll be voluntarily throwing out the one thing that nobody can take from you: your integrity. Will it stop there, or will you do it again, the next time something's not quite perfect? How much of your work will be tainted? Pretty much all of us researchers struggle with impostor syndrome, but if you go down this path, you'll know it's true. Do you really want to live that way?

Not only that, but you will have lied and compromised yourself over something really stupid, just to make a graph a little bit prettier. If you have real results, they will stand, even with noise. If the noise is big enough to actually be a problem, then that's not a problem, that's an opportunity. As the quote attributed to Asimov goes:

The most exciting phrase to hear in science, the one that heralds new discoveries, is not “Eureka” but “That’s funny...”

A lot of important emergent phenomena in computer science get discovered that way as well. If you lie, not only are you compromising your integrity and risking total damnation if it ever gets discovered, but you are also cutting off the possibility that you might stumble over something more important than what you were doing at first.

In short: don't do it.

Answer 4

Why not run the experiment enough times so that you can produce your plot with error bars on the points? This will make it possible for the reader to understand how much random variation there is in the measurements.

Answer 5

Others have provided useful input but I am not sure they completely addressed the “How can editors and reviewers detect data manipulation?” question.

The simple answer is that mostly, they can't and they don't, certainly not in fields where researchers don't routinely share code, raw data, photographs and the like but only statistical tests or basic plots. If you are really sloppy, you might end up with incoherent numbers that could not possibly have been produced by the analysis you claim to have done (I have seen things like that) but more subtle manipulation is not so easy to detect.

There are a few fascinating techniques to detect bogus data (including but not limited to Benford's law) but very few people actually have the expertise required and reviewers do not routinely check for that. In most cases, such an analysis can give you a strong presumption but no solid proof. Some famous data sets have been thoroughly analyzed without reaching a consensus (e.g. Cyril Burt's work on intelligence and heredity).

If you look at some of the high profile cases of fraud exposed in recent years (Jens Förster but also Diederik Stapel or Dirk Smeesters), they were mostly found out after many many fraudulent publications and not always because there was anything suspicious about these publications. The more “greedy” the fraudster is, the clearer the pattern becomes and some people might have had private misgivings at some stage but the fraud is only exposed later, usually after someone blew the whistle and not because a reviewer noticed it.

You can look at this as a glass half full (fraud is eventually detected) or half empty (How could it go on for so long? How many others are out there?) but the fact is that it's only in the aggregate that the results look suspicious, not at the level of a single graph or article.

Not that I advocate doing that, of course. Ethically, it's clearly wrong and the cases I just mentioned show that you can get found out in other ways and face very serious consequences. But reviewers and editors usually can't detect fraud directly, that's not how the systems works.

Answer 6

At the point at which you only have the figure, or the underlying processed data, you cannot detect "well crafted" manipulation. One aspect of reproducible research, which is becoming more popular, requires that others be able to reproduce the data. This means making code available, describing hardware in sufficient detail, and also proving things like seeds and states of random number generators. This allows reviewers to recreate your data and then test how sensitive they are to slight perturbations.