Should I recommend rejection for a paper I referee if the authors don't make changes that they could have made?

Question

Earlier this year, I received a paper to referee for the top journal in my field of research. I know both of the authors. The first recently completed a Ph.D. working with a friend of mine. My faculty friend is the best-known person working in our area; he is an extremely careful and conscientious researcher, and working under him, the student produced some very good work. The other author of the manuscript I'm refereeing is the new post-doctoral supervisor of that recent Ph.D. grad. He is a very senior but not terribly distinguished scientist, and I think that some of his work is rather slipshod.

The first manuscript draft I got from this pair of authors had a glaring problem. There may have been a fundamental error in how they interpreted their results. There are two different ways that the system they are studying could behave, and they assume that it goes one way, apparently without even noticing the other. They might well be right about how the system behaves, but I sent back a report saying that the work would be publishable if they either explained why they only considered one of the two possibilities or expanded their analysis to cover both. The final conclusion is likely to unchanged whichever way the system behaves, but it is important to verify this; moreover, it is interesting in its own right to know which way things go.

After a rather brief period, I got a revised manuscript back from the journal. The authors had made a number of other minor corrections that I asked for, but they basically ignored my main point. At this point, I'm not sure what to do. I still feel like the paper contains a significant amount of interesting material, and it could be fixed without that much effort. However, I am upset that the authors made not effort to fix the actual problem I pointed out. They don't even really acknowledge the problem in their resubmission letter.

I was the only referee who turned in a report on the first draft, so what I say will almost certainly determine whether the paper is published. Should I send it back one more time, insisting that the changes I called for actually be made this time? Or should I recommend rejection, since the authors are apparently unwilling to make reasonable adjustments.

Answer 1

It's totally reasonable to be upset that the authors ignored your comments, but for the purposes of reviewing the paper, I think you should set that aside. I would let the editor make the decision as to whether the authors are "unwilling to make reasonable adjustments". Send him/her a note saying something like:

I mentioned in my initial report that it was essential for the revisions to address the issue of XXXX, but this has not been done, nor mentioned in the authors' comments. This paper has many good qualities, but I do not think the paper is suitable for publication if this issue is not addressed. If you feel it is worth the time to ask the authors for a further revision addressing this issue, I would be happy to read it and give my opinion.

I think it's up to the editor to judge between "this is a paper I really want in my journal, so I'm willing to put up with a little bit of crap" or "the authors are wasting our time, to hell with them".

Answer 2

If it is indeed the case that:

1. The potential flaw really could fundamentally change the conclusion of the paper, and
2. The authors didn't even address your issue at all even in their response letter

then I would recommend rejection.

When doing a major revision, it is imperative that the authors at least speak to every point raised by the reviewers in the response letter, even if it is just to say: "No, we will not do this."

If an author blatantly fails to address a critical issue, then I feel that it is reasonable to conclude that they are avoiding the issue and recommend rejection. If the issue is important but not critical, or if you disagree with how the authors address it, then I think it is better instead to recommend major revision.

Even though you recommend rejection, if the editor wants to give them another chance, they can certainly do so. Similarly, even if they are rejected, they can amend their ways (or not) and go to another lower-level journal. So: vote your scientific conscience.

Answer 3

Should I send it back one more time, insisting that the changes I called for actually be made this time? Or should I recommend rejection, since the authors are apparently unwilling to make reasonable adjustments.

As long as they don't even react to the remark, the polite move would be to assume a good faith communication problem, especially given the amount of involved people. So I'd just ask for another revision and point out the remark was not addressed anyhow.

Personally, I'd avoid being subtle once they missed the point, so I'd just spell out that the authors should address the remark (in the paper), convince you they shouldn't in the resubmission letter, or you will recommend rejection. To avoid being too negative or aggressive, I'd also put in positive comments — say, you expect the point to be easily addressed.

Answer 4

You should encourage the editor to get a second opinion. If you feel strongly, you should reject the paper again citing or repeating your comment about the alternate explanation. Based on what you've said here, it seems likely to be important.

Answer 5

It seems to me that you should write a brief review which starts by saying something like "I recommend rejecting the submission X, on the grounds that the paper fails to even consider a plausible alternative interpretation of the experimental results, and thus the theoretical significance of the paper is indeterminate". In my review, I would reiterate the essential points about the alternative made in the earlier review, and I would also state this as an "I already mentioned this" summary. It is scientifically not acceptable to claim or imply that a specific conclusion has been established based on some evidence, when in fact multiple conclusions are consistent with that evidence.

It is not clear whether your previous recommendation was "revise and resubmit" or "reject" -- I assume it was "revise and resubmit", otherwise it would be strange for the paper to have been revised and resubmitted. I think therefore that it would be useful to explain to the authors and the editor the rationale for a more strongly negative recommendation, when the paper has not (apparently) gotten worse. A number of journals use the "one round of revision" standard, where you judge that after one revision, the paper is likely to be acceptable. Under that standard, if you judge that a paper is likely to be acceptable after one revision, it should be rejected. A corollary is that if a paper turns out to not be acceptable after a single revision, it should be rejected. Now, I do know of journals where 4 rounds of revisions are tolerated, so you may not be able to contextualize the "reject" decision based on announced journal policy, but you can certainly adopt such a standard, implicitly or explicitly, as your own quality standard for acceptance.