18

This is a follow-up to this question: Should I recommend rejection for a paper I referee if the authors don't make changes that they could have made?

To recap: I was sent a paper to referee by a top journal. My first report pointed out that the authors had only considered one of two possible behavior patterns that their system might have. When the authors resubmitted, they basically ignored what I had said (although see below). So I sent it back, asking that the changes I had asked for before actually be made.

When I read the second draft very carefully, I saw they had actually added a very short paragraph that seemed to be arguing against the second type of behavior I was worried about being possible. It was so minor I didn't even notice it before I asked my earlier StackExchange question. However, the manuscript authors made an error in their attempt to demonstrate this point quantitatively; as I pointed out in my second report, their argument actually showed that I was correct.

Now, they have submitted the paper a third time. This time, they very enthusiastically acknowledge that I am right--both types of system behavior are expected to occur. However, they now claim that their numerical calculations actually include both possibilities, and that they actually did all along! The letter says that they just explained things really badly.

They have made more changes to the manuscript. It now states that both behaviors can occur and that both are fully included. I am not in a position to check whether their results are consistent with this. The paper does a lot of heavy numerical calculation, which would take a long time to replicate. Since the bottom line result is not going to depend much on the intermediate details, it is plausible that their graphs could have been showing the fully correct answers all along.

However, the format of the paper suggests otherwise. Add to that the fact that the second draft of the paper actually argued obliquely against the type of behavior that they now admit is natural. I feel like there are two likely possibilities. The first is that the authors are being dishonest. They missed the second possibility, and now that I have pointed out that it definitely occurs, they have reasoned that they can fib about the matter and it will be difficult to refute their claim. I am rather unhappy about this possibility; I hope it's not the case. The second possibility is that the communication between the two authors of the paper was extremely poor. The person who did the actual writing would have to have been quite ignorant of what his coauthor was actually calculating. This is a less troubling but still far from ideal situation.

So, once again, I find myself unsure what to do. Should I share my misgivings with the editor? Or should I take the current draft's claims as is--chalk the whole thing up to incompetence rather than dishonesty? I have, after all, no concrete evidence of fraud.

  • 9
    You haven't been able to check all their calculations, you can't be sure their explanation is wrong, you're drawing conclusions based on the format of the paper, and your first thought is that the authors may be behaving dishonestly. I think you're jumping to conclusions. I suggest you step back a little bit and focus more on the substance of the paper and less on the authors' motivations. Never ascribe to malice that which can adequately be explained by incompetence/sloppiness/miscommunication/inadvertent error/oversight. They explicitly said they explained things badly -- it happens! – D.W. Jun 6 '15 at 0:32
  • 6
    Can I ask whether you think there is dishonesty in the new draft of the paper or just via the discrepancy between the way the authors are presenting their knowledge from the first draft to the current draft? To me this is an important distinction: "I knew it all along" may be an instance of dishonesty by an academic but not an instance of academic dishonesty. Almost every person I've ever met tries to frame themselves in the best possible light, to the extent of taking liberties with objective reality. If this bending affects the academic work itself: problem. If it doesn't: problem? – Pete L. Clark Jun 6 '15 at 2:08
  • 3
    @D.W. If you haven't been able to check the results (for whatever reason), should you ever recommend accepting the paper? – Raphael Jun 6 '15 at 7:14
  • 7
    "However, they now claim that their numerical calculations actually include both possibilities, and that they actually did all along!" ...what exactly do you see wrong with this? Initially they interpreted the data one way, and now a different way. And they didn't even say "we knew it all along"; they said "our results supported this all along", which seems like exactly what you (as a reviewer) would want to see if their data were obtained properly. What are you really seeking in their paper at this point? Are you upset they haven't given you credit or something? – user541686 Jun 6 '15 at 11:15
  • 4
    Maybe the numerical calculations did always include the other behaviour, but they hadn't realised this until you convinced them of its importance. Having realised this, they have updated the paper accordingly. In which case everything is fine and correct and the peer review process is working as intended. – Nathaniel Jun 7 '15 at 2:25
11

I have found myself in similar situations in the past. However, in such cases, the fraudulent behavior was much more obvious. This is a bit of a mixed case.

So long as the results are plausible, the authors do have some liberty to do what you say they have. However, given that they should have also known that the same referees would review the paper again, the way they have addressed the changes does seem to be in poor taste.

I would recommend mentioning what you find problematic in your comments— confidentially to the editor if necessary, but preferably in the "open" comments the authors get to see. This will make it clear exactly why you believe the paper should (or should not) be accepted. You could state something like the following:

Scientifically the results are acceptable, but I am highly uncomfortable with and concerned by the way in which the authors have incorporated the changes. The editor should take these issues into consideration before rendering a final decision.

That puts the decision in the editor's hands (where it rightfully belongs), while you get to make whatever case you feel is appropriate.

| improve this answer | |
  • 1
    I've no clue why is the other answer who got all the upvotes, this one seems reasonable and to the point, while the other one basically says "you suspect something, don't bother, go on, everything's nice" which is a wtf for me. – o0'. Jun 6 '15 at 7:31
  • 1
    @Lohoris I addressed your comment in my answer. Maybe this makes it clearer what I meant. – xLeitix Jun 6 '15 at 8:45
47

It is hard to tell from just your description alone (and without knowing the actual paper and revisions), but honestly this does not really sound like unethical behavior to me. Basically, what seemed to have happened was the following:

  1. Authors hand the paper in. You review and claim that one of two important aspects is missing.
  2. The authors disagree and, in an attempt to calm you over, write a short token paragraph about the second aspect (that they, in all honesty, don't believe to be very important).
  3. You insist that the second aspect is super-important and needs to be covered more explicitly.
  4. The authors give up and describe what the may think to be a minor aspect in much more detail than what they originally planned to do. Note that it is well possible that the authors were fully aware of this and had the data to do this all the time, but just did not think about it as something important. That they now agree that this is super-important sounds more like an attempt to win you over than anything else.

I have seen these kinds of exchanges play out in many, many journal submissions. Whether this actually improves the paper or not depends primarily on whether the second aspect was actually important or not. You say it is, but I have definitely seen reviewers get hung up on absolute sidenotes as well (which I am sure they honestly thought were essential), ultimately to the detriment of the manuscript.

More generally, it sounds a bit like you were a bit the "damned if you do, damned if you don't" kind of reviewer in this process. What kind of change did you expect after the second review round? You wanted this additional behavior covered, and they did - and now that they did, you wonder why they did not do so from the beginning and presume that something unethical is happening. If they wouldn't have addressed this point, you would have been unhappy because the authors did not fix your major comments.

What I generally try to teach my students is that when you review for journals, you need to recommend fixes that can actually be done. That is, there needs to be a reasonable and doable change that would actually make you happy to accept the manuscript. If you think that a paper can in no reasonable way be fixed, recommend rejection (and don't accept another review if the editor decides for revision). Never recommend revision if you won't like any outcome.

Edit based on Lohoris' comment to aismail's answer:

I've no clue why is the other answer who got all the upvotes, this one seems reasonable and to the point, while the other one basically says "you suspect something, don't bother, go on, everything's nice" which is a wtf for me.

Accusing authors of fabricating the data is a serious suspicion and should not be raised lightly if there are other plausible explanations. It's not that you "shouldn't bother", it's that it simply does not sound like the data has been fabricated to me (or, at least, it does not sound significantly more likely than in any journal submission where the code etc. is not public, which is a systemic problem). There are any number of (good or at least less bad) reasons why the authors may now have data that they did not report on in the first version, including:

  • The authors generated this data in the many months between revisions
  • The authors had the data all along but did not consider it very important
  • The authors wanted to save up this data for another paper (either to salami-slice, or because they truly thought that those should logically be two separate papers)

It is widely accepted that even major changes to the data and analysis can happen in between major revisions, and indeed it is common-place at least in my community to add more data to the paper based on reviewer request. This is basically the reason why we do peer review in the first place - if we always remain suspicious when the authors do more than cosmetic changes, why do we recommend revisions in the first place?

I understand that the authors were talking about a "miscommunication" in the response letter, but frankly, after two revisions (especially with a very critical reviewer in the loop), most authors tend to write in the response letter whatever they think gives them the best chance to calm the reviewer over. I would not give too much weight to whatever people write in this non-public document.

| improve this answer | |
  • 3
    I don't disagree with your last paragraph, but the problem in this case is that the OP didn't know whether the fix could be done: the paper failed to consider a certain possibility, and the OP didn't know whether this was because that possibility could easily be dispensed with, vs. whether this was a fatal flaw that potentially invalidated the paper's whole conclusion. What do you recommend in that sort of situation? – ruakh Jun 6 '15 at 7:17
  • I'm afraid that you have not entirely understood the situation. I'm sure the misunderstanding is my fault, since I have been rather elliptical in my statements, not wanting to reveal too much about the scientific content of the paper; moreover, the history is spread over two separate questions. Let me try to clarify: The first draft only considered one possible behavior pattern for the system. Anybody knowledgeable about the field would know that if the second behavior patter occurred, it would be important. My conclusion is that the authors have entirely overlooked the second possibility – Buzz Jun 6 '15 at 20:06
  • ... After I pointed this out in my first report, they did not address the point at all in their response letter. However, I discovered they had added a section trying to show the second type of behavior was impossible. Yet their calculation contained an error, and it in fact showed the second behavior was extremely likely! The third draft acknowledged this, but it claimed that the numerical calculations had always included both possibilities. Either they are being dishonest, or the person doing the calculations is not communicating with the person writing the explanations. – Buzz Jun 6 '15 at 20:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.