How do I referee a paper when the authors may be behaving dishonestly?

Question

This is a follow-up to this question: Should I recommend rejection for a paper I referee if the authors don't make changes that they could have made?

To recap: I was sent a paper to referee by a top journal. My first report pointed out that the authors had only considered one of two possible behavior patterns that their system might have. When the authors resubmitted, they basically ignored what I had said (although see below). So I sent it back, asking that the changes I had asked for before actually be made.

When I read the second draft very carefully, I saw they had actually added a very short paragraph that seemed to be arguing against the second type of behavior I was worried about being possible. It was so minor I didn't even notice it before I asked my earlier StackExchange question. However, the manuscript authors made an error in their attempt to demonstrate this point quantitatively; as I pointed out in my second report, their argument actually showed that I was correct.

Now, they have submitted the paper a third time. This time, they very enthusiastically acknowledge that I am right--both types of system behavior are expected to occur. However, they now claim that their numerical calculations actually include both possibilities, and that they actually did all along! The letter says that they just explained things really badly.

They have made more changes to the manuscript. It now states that both behaviors can occur and that both are fully included. I am not in a position to check whether their results are consistent with this. The paper does a lot of heavy numerical calculation, which would take a long time to replicate. Since the bottom line result is not going to depend much on the intermediate details, it is plausible that their graphs could have been showing the fully correct answers all along.

However, the format of the paper suggests otherwise. Add to that the fact that the second draft of the paper actually argued obliquely against the type of behavior that they now admit is natural. I feel like there are two likely possibilities. The first is that the authors are being dishonest. They missed the second possibility, and now that I have pointed out that it definitely occurs, they have reasoned that they can fib about the matter and it will be difficult to refute their claim. I am rather unhappy about this possibility; I hope it's not the case. The second possibility is that the communication between the two authors of the paper was extremely poor. The person who did the actual writing would have to have been quite ignorant of what his coauthor was actually calculating. This is a less troubling but still far from ideal situation.

So, once again, I find myself unsure what to do. Should I share my misgivings with the editor? Or should I take the current draft's claims as is--chalk the whole thing up to incompetence rather than dishonesty? I have, after all, no concrete evidence of fraud.

Answer 1

I have found myself in similar situations in the past. However, in such cases, the fraudulent behavior was much more obvious. This is a bit of a mixed case.

So long as the results are plausible, the authors do have some liberty to do what you say they have. However, given that they should have also known that the same referees would review the paper again, the way they have addressed the changes does seem to be in poor taste.

I would recommend mentioning what you find problematic in your comments— confidentially to the editor if necessary, but preferably in the "open" comments the authors get to see. This will make it clear exactly why you believe the paper should (or should not) be accepted. You could state something like the following:

Scientifically the results are acceptable, but I am highly uncomfortable with and concerned by the way in which the authors have incorporated the changes. The editor should take these issues into consideration before rendering a final decision.

That puts the decision in the editor's hands (where it rightfully belongs), while you get to make whatever case you feel is appropriate.

Answer 2

It is hard to tell from just your description alone (and without knowing the actual paper and revisions), but honestly this does not really sound like unethical behavior to me. Basically, what seemed to have happened was the following:

1. Authors hand the paper in. You review and claim that one of two important aspects is missing.
2. The authors disagree and, in an attempt to calm you over, write a short token paragraph about the second aspect (that they, in all honesty, don't believe to be very important).
3. You insist that the second aspect is super-important and needs to be covered more explicitly.
4. The authors give up and describe what the may think to be a minor aspect in much more detail than what they originally planned to do. Note that it is well possible that the authors were fully aware of this and had the data to do this all the time, but just did not think about it as something important. That they now agree that this is super-important sounds more like an attempt to win you over than anything else.

I have seen these kinds of exchanges play out in many, many journal submissions. Whether this actually improves the paper or not depends primarily on whether the second aspect was actually important or not. You say it is, but I have definitely seen reviewers get hung up on absolute sidenotes as well (which I am sure they honestly thought were essential), ultimately to the detriment of the manuscript.

More generally, it sounds a bit like you were a bit the "damned if you do, damned if you don't" kind of reviewer in this process. What kind of change did you expect after the second review round? You wanted this additional behavior covered, and they did - and now that they did, you wonder why they did not do so from the beginning and presume that something unethical is happening. If they wouldn't have addressed this point, you would have been unhappy because the authors did not fix your major comments.

What I generally try to teach my students is that when you review for journals, you need to recommend fixes that can actually be done. That is, there needs to be a reasonable and doable change that would actually make you happy to accept the manuscript. If you think that a paper can in no reasonable way be fixed, recommend rejection (and don't accept another review if the editor decides for revision). Never recommend revision if you won't like any outcome.

Edit based on Lohoris' comment to aismail's answer:

I've no clue why is the other answer who got all the upvotes, this one seems reasonable and to the point, while the other one basically says "you suspect something, don't bother, go on, everything's nice" which is a wtf for me.

Accusing authors of fabricating the data is a serious suspicion and should not be raised lightly if there are other plausible explanations. It's not that you "shouldn't bother", it's that it simply does not sound like the data has been fabricated to me (or, at least, it does not sound significantly more likely than in any journal submission where the code etc. is not public, which is a systemic problem). There are any number of (good or at least less bad) reasons why the authors may now have data that they did not report on in the first version, including:

• The authors generated this data in the many months between revisions
• The authors had the data all along but did not consider it very important
• The authors wanted to save up this data for another paper (either to salami-slice, or because they truly thought that those should logically be two separate papers)

It is widely accepted that even major changes to the data and analysis can happen in between major revisions, and indeed it is common-place at least in my community to add more data to the paper based on reviewer request. This is basically the reason why we do peer review in the first place - if we always remain suspicious when the authors do more than cosmetic changes, why do we recommend revisions in the first place?

I understand that the authors were talking about a "miscommunication" in the response letter, but frankly, after two revisions (especially with a very critical reviewer in the loop), most authors tend to write in the response letter whatever they think gives them the best chance to calm the reviewer over. I would not give too much weight to whatever people write in this non-public document.