48

I reviewed a paper submitted for a smallish magazine. It presented an algorithm to perform some allocation task and compared its performance to that of several other algorithms from the literature performing the same task in several ways (the results, i.e. the allocation, can be evaluated based on the usage of several different resources, so a result could use less of resource A, but more of resource B, and so on).

My opinion was that, although the algorithm was badly presented and the paper was nigh-incomprehensible, the results presented seemed good, so the authors deserved another shot at better explaining themselves, so I did not suggest to reject it altogether.

The first review round went through with a unanimous "major revisions" verdict.

Then I was asked to review the second submitted version of the paper too. In this new version, the algorithm had been compared to a much broader range of algorithms. Problem is: even though the algorithms it was being compared to changed, the comparison charts remained exactly the same, and looking at them side-by-side revealed no difference whatsoever (no explicit numerical data was provided).

What is worse is that the change was not even one-to-one. In the first submission, the algorithm (let's call it A) was compared with the same three other algorithms in all categories (resource A usage, resource B usage etc.) while in the second submission, each resource comparison involved different algorithms, so for example, A was compared to B,C and D in resource A utilization, but it was compared to C, E and F in resource B utilization, and so on.

Nonetheless, each chart in the second submission was identical to one from the first submission.

At this point, I was fairly certain that at least the second round of comparisons had been completely faked, i.e. the authors just changed the labels on the charts.

Asking one of my senior coworkers, I was advised to just ignore the issue and to not raise a ruckus, since this issue has a high chance of backfiring: we are not an academic institution, we are the R&D department of a pretty small private firm, hence we have very little political weight and scientific reputation.

I am wondering if I really should raise this issue with the editor, with whom my firm has business relations, as we are partners in several government-funded projects, or I should heed the advice of my colleague.

While the paper has very little chance of being published as the second submission is also nigh-unreadable, a co-author of this paper has an extremely high h-index (100+), hence I feel if my suspicion is founded, it really should be brought to the light.

  • 70
    I don't understand how reviewing a paper can backfire. It seems irrelevant that you work for "a pretty small private firm, [with] very little political weight and scientific reputation." Nor do I understand why having business relations with the editor should influence your review. – user2768 Dec 8 '17 at 15:12
  • 46
    Re: user2768's comment: Conversely, I would think if there is any political/relationship damage to be done, it could be equivalently done by you letting known errors/fraud slide through, when the editor was partly relying on your help for the review. – Daniel R. Collins Dec 8 '17 at 15:38
  • 20
    I agree with @DanielR.Collins here -- you risk just as much damage by letting a potentially fraudulent paper through. By asking for more information, you appear (and are being) thorough; better this than viewed as uninformed or careless. – deckeresq Dec 8 '17 at 16:36
  • 10
    This kind of discrepancy seems like it could arise unintentionally; is it possible they were rushing out the revised paper and simply mislabelled the charts by accident? – Dan Staley Dec 8 '17 at 17:31
  • 5
    @MarcoCapitani If you're going to let such a thing pass, why even bother reviewing at all? If you don't have the power to point out significant problems, reviewing seems to be a particularly pointless waste of your time. – sgf Dec 9 '17 at 21:31
122

You should definitely report your concern, but assume good faith.

This certainly sounds like a significant problem. However, fabricating data is a very serious (even career-ending) allegation. You shouldn't accuse someone of this without very strong evidence, and I don't think you have it in this case. There could be an innocent explanation.

  • You might have misunderstood what they are doing. For example, rather than running all new tests, did they use the same results for their algorithm, but then compare it to different algorithms? (I'm not sure from the question whether this would be possible).
  • It could be a simple error. For example, what if they accidentally opened the wrong image file and added labels to it?

I would raise the issue, but rather than saying "this looks faked", something like this:

The authors purport to have run new comparisons, and yet the results on the graphs are exactly the same as in their previous draft. I don't understand how this can be correct. Could they please explain, or correct the graphs if necessary?

Another thing you should do is ask for more detailed results and more information about their methods. It sounds like their reporting on what they have done is far from adequate. How they respond to this request might give more evidence on whether the results might be faked. If they are unable to convincingly explain the strange results and fully describe their methods, you should then at least raise the concern with the editor. I don't think you have enough cause to do that yet, though.

If there is further action to be taken, it will be the editor's responsibility. Here is what the Committee on Publication Ethics recommends to editors in this situation. If the authors cannot satisfactorily explain themselves, it should result in a report to their institution and an investigation.

  • 40
    This. I'm sure I'm not the only one who has almost pasted the wrong chart/table/attachment into an important document. – shoover Dec 8 '17 at 21:00
  • 2
    Yes; the concern is valid, but it's more likely to be an error than deliberate. – Joe McMahon Dec 8 '17 at 21:53
  • 4
    If they are unable to convincingly explain the strange results and fully describe their methods, do you even need to raise the concern of faking data? It seems like that alone would be enough to warrant rejection. – jpmc26 Dec 9 '17 at 2:49
  • 3
    Even more diplomatic: "It appears they made a mistake and put the new labels on the old comparison graphs." – Loren Pechtel Dec 9 '17 at 3:12
  • 1
    @jpmc26 fabricating data ought to have consequences beyond rejection. Practically, that may not be possible, but it would be up to the editor to take further action, so I think informing the editor is due diligence in a case where there is strong reason to suspect fraud. – user24098 Dec 9 '17 at 6:48
23

Most (all?) peer review processes allow you to write a private note to the editor that isn’t shown to the paper authors. Use this to raise your concern with the editor, providing a detailed explanation of the evidence.

As for the public part of the review, it’s entirely legitimate to note that the description in the paper is insufficient to reproduce the results (which it seems to be, from your description): if the data isn’t faked, the authors should have no issue describing the method in sufficient detail that the reader is able to recapitulate it completely.

In fact, your description of the vague results in the paper alone would be grounds to demand an appropriate revision.

To address your senior coworker’s comment: they are wrong. Data fabrication is a serious breach of research ethics. As a reviewer, you mustn’t let it slide under any circumstances — regardless of rejection status of the manuscript.

0

Unfortunately, you have a much better chance of bullying someone without any repercussions whatsoever as an anonymous journal-referee than as the most lousy moderator of academia.stackexchange. ;-) More seriously: no worries, simply go for criticism if your report is anonymous. If you report is not anonymous (which is seldom), do the same thing as you do when writing a good-looking recommendation letter which is actually "a little questionable" in its actual contents.

In any case, you’ve done a much better review than the overwhelming majority of the research-level referees already.

  • 2
    I suggest you remove the passage about stackexchange mods from your answer. I don't see how it is related to the question. – Dmitry Grigoryev Dec 11 '17 at 9:15
0

I have contributed as a reviewer to over forty papers to date. In most cases, I suggested (or “hinted”, because whenever possible I prefer not telling the editor what to do) rejecting the paper. I have signed a third of my reviews, but I only signed a negative review once. I am striving to sign my reviews more often.

I understand there can be some backlash. More often than expected. Unfortunately too many editors nowadays have ties with authors, and some authors just cannot take being criticised. The main reason I do not sign all of my reviews is fear of backlash, especially because I am still a postdoc. Thus said, find my recommendation below.

Raise the issue of duplicated images openly, and emphasise on the fact that the submission had already deep issues in its original form. Comment on any other facets of the new submission and finish your review coldly. If you are forced by the system to suggest a recommendation, chose rejection. I believe you have all the reason to reject a paper which hasn’t improved significantly when given the chance, and there should be no ruckus over that. Just do not come around accusing authors of misconduct over duplicated images because sloppy behaviour is not fraud. If you just must, merely hint some concern over the issues to the editor in private, but in your place I'd avoid that because of potential side conflicts that you mentioned.

0

"the second submission is also nigh-unreadable"

This in itself seems like a reason to reject it totally.

Either:

  1. It is unreadable garbage (Reject NOW!)
  2. The subject matter could be too complex for you to understand.

I would try and ask other about the contents of the rest of the paper to see if anyone understand any of it. Find some skilled in that area of research.

Authors failed to understand the definition of "major revisions", send back for more major revisions.

If you let garbage through you will never by scientifically respected.

  • 1
    This sounds like a recommendation to find an excuse to reject the paper, instead of voicing the OP's real concerns. – Dmitry Grigoryev Dec 11 '17 at 9:12
0

Considering that the paper describes the algorithm, by definition, the author isn't hiding it... so why not just require the author to send you the source code, not binaries, but the actual source, which you can build, run, and thus create the data yourself?

The source should include not only his algorithm but also all other algorithms that were used in the paper for comparison.

Then compare the data you got from running his code with the data in the paper and see if there are any differences.

IMO this should be a standard for any paper about algorithms - submit the source code that anyone can build and run and prove it.

You can also set up a standard regarding what is and isn't allowed to be used in the source, so that you (or anyone else) can easily build it without needing proprietary libraries. For example, "VS 2017 Community Edition - C++ or C#" or Python or Java will probably cover 99% of such cases.

-10

That is why we have reproducibility requirement in science. Whatever was measured must be verifiable independently by others. Over and over and over...

But if the claims don't get accepted for publication anywhere it won't even be known to the community what to try to verify or to disprove. Imagine Mickelson-Morleys paper of the measurement of the speed of light being refused everywhere for example. Would clearly be no good.

  • 5
    Sorry, it's an American colloquialism, I'm not actually guessing. What I mean is, the logical conclusion of your answer appears to be that peer review should not exist at all. If an experiment being faked is not reason to prevent publication, then what possible reason could there be? – user24098 Dec 9 '17 at 9:22
  • 5
    I agree, and I literally started my answer with "assume good faith" in the first sentence. But I think they need to back up the strange data with evidence before it gets published. If they can fully document how they got their results, then yes, let it be published even if the results are hard to explain. But your answer appears to imply it should just be accepted for publication without any further checks. – user24098 Dec 9 '17 at 9:44
  • 5
    The OP asks "what should I do as a reviewer?" and you respond "the work needs to get published so others can verify it" (without qualifying that in any way). What conclusion is one supposed to draw from that? – user24098 Dec 9 '17 at 9:53
  • 4
    @mathreadler This is Stack Exchange, not science. We're interested in answering the question at the top of the page, not arguing about how science should be done. – David Richerby Dec 9 '17 at 12:06
  • 8
    Reproducibility and verification require a published paper to adequately explain how results were generated, which is precisely what I am advocating here. – user24098 Dec 9 '17 at 12:25

protected by Alexandros Dec 12 '17 at 19:19

Thank you for your interest in this question. Because it has attracted low-quality or spam answers that had to be removed, posting an answer now requires 10 reputation on this site (the association bonus does not count).

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.