I have a strong suspicion and weak proof that the data in the paper are fake. What to do?

Question

I reviewed a paper submitted for a smallish magazine. It presented an algorithm to perform some allocation task and compared its performance to that of several other algorithms from the literature performing the same task in several ways (the results, i.e. the allocation, can be evaluated based on the usage of several different resources, so a result could use less of resource A, but more of resource B, and so on).

My opinion was that, although the algorithm was badly presented and the paper was nigh-incomprehensible, the results presented seemed good, so the authors deserved another shot at better explaining themselves, so I did not suggest to reject it altogether.

The first review round went through with a unanimous "major revisions" verdict.

Then I was asked to review the second submitted version of the paper too. In this new version, the algorithm had been compared to a much broader range of algorithms. Problem is: even though the algorithms it was being compared to changed, the comparison charts remained exactly the same, and looking at them side-by-side revealed no difference whatsoever (no explicit numerical data was provided).

What is worse is that the change was not even one-to-one. In the first submission, the algorithm (let's call it A) was compared with the same three other algorithms in all categories (resource A usage, resource B usage etc.) while in the second submission, each resource comparison involved different algorithms, so for example, A was compared to B,C and D in resource A utilization, but it was compared to C, E and F in resource B utilization, and so on.

Nonetheless, each chart in the second submission was identical to one from the first submission.

At this point, I was fairly certain that at least the second round of comparisons had been completely faked, i.e. the authors just changed the labels on the charts.

Asking one of my senior coworkers, I was advised to just ignore the issue and to not raise a ruckus, since this issue has a high chance of backfiring: we are not an academic institution, we are the R&D department of a pretty small private firm, hence we have very little political weight and scientific reputation.

I am wondering if I really should raise this issue with the editor, with whom my firm has business relations, as we are partners in several government-funded projects, or I should heed the advice of my colleague.

While the paper has very little chance of being published as the second submission is also nigh-unreadable, a co-author of this paper has an extremely high h-index (100+), hence I feel if my suspicion is founded, it really should be brought to the light.

Answer 1

You should definitely report your concern, but assume good faith.

This certainly sounds like a significant problem. However, fabricating data is a very serious (even career-ending) allegation. You shouldn't accuse someone of this without very strong evidence, and I don't think you have it in this case. There could be an innocent explanation.

• You might have misunderstood what they are doing. For example, rather than running all new tests, did they use the same results for their algorithm, but then compare it to different algorithms? (I'm not sure from the question whether this would be possible).
• It could be a simple error. For example, what if they accidentally opened the wrong image file and added labels to it?

I would raise the issue, but rather than saying "this looks faked", something like this:

The authors purport to have run new comparisons, and yet the results on the graphs are exactly the same as in their previous draft. I don't understand how this can be correct. Could they please explain, or correct the graphs if necessary?

Another thing you should do is ask for more detailed results and more information about their methods. It sounds like their reporting on what they have done is far from adequate. How they respond to this request might give more evidence on whether the results might be faked. If they are unable to convincingly explain the strange results and fully describe their methods, you should then at least raise the concern with the editor. I don't think you have enough cause to do that yet, though.

If there is further action to be taken, it will be the editor's responsibility. Here is what the Committee on Publication Ethics recommends to editors in this situation. If the authors cannot satisfactorily explain themselves, it should result in a report to their institution and an investigation.

Answer 2

Most (all?) peer review processes allow you to write a private note to the editor that isn’t shown to the paper authors. Use this to raise your concern with the editor, providing a detailed explanation of the evidence.

As for the public part of the review, it’s entirely legitimate to note that the description in the paper is insufficient to reproduce the results (which it seems to be, from your description): if the data isn’t faked, the authors should have no issue describing the method in sufficient detail that the reader is able to recapitulate it completely.

In fact, your description of the vague results in the paper alone would be grounds to demand an appropriate revision.

To address your senior coworker’s comment: they are wrong. Data fabrication is a serious breach of research ethics. As a reviewer, you mustn’t let it slide under any circumstances — regardless of rejection status of the manuscript.

Answer 3

Unfortunately, you have a much better chance of bullying someone without any repercussions whatsoever as an anonymous journal-referee than as the most lousy moderator of academia.stackexchange. ;-) More seriously: no worries, simply go for criticism if your report is anonymous. If you report is not anonymous (which is seldom), do the same thing as you do when writing a good-looking recommendation letter which is actually "a little questionable" in its actual contents.

In any case, you’ve done a much better review than the overwhelming majority of the research-level referees already.

Answer 4

I have contributed as a reviewer to over forty papers to date. In most cases, I suggested (or “hinted”, because whenever possible I prefer not telling the editor what to do) rejecting the paper. I have signed a third of my reviews, but I only signed a negative review once. I am striving to sign my reviews more often.

I understand there can be some backlash. More often than expected. Unfortunately too many editors nowadays have ties with authors, and some authors just cannot take being criticised. The main reason I do not sign all of my reviews is fear of backlash, especially because I am still a postdoc. Thus said, find my recommendation below.

Raise the issue of duplicated images openly, and emphasise on the fact that the submission had already deep issues in its original form. Comment on any other facets of the new submission and finish your review coldly. If you are forced by the system to suggest a recommendation, chose rejection. I believe you have all the reason to reject a paper which hasn’t improved significantly when given the chance, and there should be no ruckus over that. Just do not come around accusing authors of misconduct over duplicated images because sloppy behaviour is not fraud. If you just must, merely hint some concern over the issues to the editor in private, but in your place I'd avoid that because of potential side conflicts that you mentioned.

Answer 5

"the second submission is also nigh-unreadable"

This in itself seems like a reason to reject it totally.

Either:

1. It is unreadable garbage (Reject NOW!)
2. The subject matter could be too complex for you to understand.

I would try and ask other about the contents of the rest of the paper to see if anyone understand any of it. Find some skilled in that area of research.

Authors failed to understand the definition of "major revisions", send back for more major revisions.

If you let garbage through you will never by scientifically respected.

Answer 6

Considering that the paper describes the algorithm, by definition, the author isn't hiding it... so why not just require the author to send you the source code, not binaries, but the actual source, which you can build, run, and thus create the data yourself?

The source should include not only his algorithm but also all other algorithms that were used in the paper for comparison.

Then compare the data you got from running his code with the data in the paper and see if there are any differences.

IMO this should be a standard for any paper about algorithms - submit the source code that anyone can build and run and prove it.

You can also set up a standard regarding what is and isn't allowed to be used in the source, so that you (or anyone else) can easily build it without needing proprietary libraries. For example, "VS 2017 Community Edition - C++ or C#" or Python or Java will probably cover 99% of such cases.

Answer 7

That is why we have reproducibility requirement in science. Whatever was measured must be verifiable independently by others. Over and over and over...

But if the claims don't get accepted for publication anywhere it won't even be known to the community what to try to verify or to disprove. Imagine Mickelson-Morleys paper of the measurement of the speed of light being refused everywhere for example. Would clearly be no good.