30

I'm writing a referee report on a paper. This paper is interesting and, if correct, would be worth publishing. However, I believe they have a major mistake in their method which invalidates their results more or less entirely. I am 90%+ confident that this method is in error, which will affect their results in (possibly unpredictable) ways.

How should I address this in the rest of the paper? I could write the rest of the referee report assuming that they haven't made this mistake - but I think that could be a waste of my time (and their time, since it'll take me longer to do).

If a referee had this sort of problem with your paper, what would you like to happen?

Edited to add: One of the major concerns I have with just rejecting, focusing on this point (as recommended by Raghu and others), is the (time and $) cost of re-running things. Saying, "This is wrong" but not going through the rest of the paper could lead to a second referee report where they've fixed the error and re-done everything. If I then say, "Also, Experiment B doesn't mean what you think it does," re-running Experiment B may have been a waste. How much consideration of these factors do you think a referee owes the authors?

  • 11
    You can just write to the editor about this particular point. If it indeed is an error, your job is done; if it is not, then there is a serious flaw in the exposition of the article, and the editor can request the authors that they revise this before further refereeing happens. – Sana Oct 7 '16 at 4:51
  • Interesting idea, and it might be possible. Though I think given the broad scope of the journal, the editor might not feel competent to make that call without consulting the authors, which means that it's like submitting the referee report anyway. – AJK Oct 7 '16 at 5:11
  • 27
    "This paper seems to suffer from a serious flaw in X. Unless this is fixed or clarified, I would recommend rejection" sounds like a perfectly legitimate report to me. No one's interested in the validity of things that are already wrong. The editor gave it to you to review because they expect that you have the competence and expertise to make such a determination. Papers that are just wrong or horrible do not need a painstaking review of every part of the paper; determining it to be wrong/horrible is already an adequate review, as long as you judge it so in good faith. – zibadawa timmy Oct 7 '16 at 6:16
  • 3
    I'll add that it's also a good idea NOT to write anything else when you're recommending rejection (or major revision). The addition of more "fixable" points just makes it more likely the editor will make a mistake. Maybe he/she will decide "most things were fixed, it can be accepted now". Putting a single major rejection-worthy item in the report makes the process more robust. – VonBeche Oct 8 '16 at 13:43
  • 4
    @VonBeche although I agree with you on the rejection part, I think that when recommending revisions (either minor or major) you should list all of your concerns. I hate when I receive the second review with "You just fixed what I told you, but then there is a) b) c) that I did not comment in the first review and now I want you to take care of..." making the review process an infinity regression thing... – Rmano Oct 9 '16 at 10:55
56

If there's a "major mistake in their method which invalidates their results more or less entirely" it seems pretty clear that you should explain this in your review, and not bother writing much else. (This seems so obvious, I wonder if I'm misunderstanding your question.) Why on earth would you "write the rest of the referee report assuming that they haven't made this mistake," if you're quite sure that the mistake exists, and that it invalidates most of the paper? What purpose would it serve, for the authors or future readers?

  • 5
    The OP said they are 90%+ sure that it's a mistake, i.e. there's a ~10% chance that it's not. I guess the point is that it wouldn't be good to reject the paper if it isn't a mistake after all. – Nathaniel Oct 7 '16 at 8:15
  • 37
    @Nathaniel Part of the job of an academic/scientific author is to write papers that are convincing and clear; and convincing and clear papers are the kind that (good) journals publish. A paper that leaves you going "there's a 90% chance this is just plain wrong" already fails that. Plus, for all we know that 10% is just a reflection of a lack of confidence in the OP. Some people just aren't comfortable with definitively saying bad things, too. – zibadawa timmy Oct 7 '16 at 10:04
  • @zibadawatimmy agreed with all of that. (I was just commenting on my interpretation of the question, not the answer. If it was me, I'd probably write a review saying "it appears that there is a serious flaw in the paper. I might be wrong about that, but if so the paper should be revised to clarify this issue before resubmitting.") – Nathaniel Oct 7 '16 at 12:29
  • 8
    If the authors fix the alleged mistake (or demonstrate that you are wrong), the time for "minor" comments on the rest of the paper is after that point. You don't know what effect the fix-up will have on the rest of the paper until you have actually seen the revised version, so there is no point doing the reviewing work twice (and even less point in writing up a review of the parts that are unpublishable because of an earlier mistake). – alephzero Oct 7 '16 at 14:31
  • Updated the question in response to Raghu's point. I think there is a real question here about how much consideration referees owe authors on this sort of issue. – AJK Oct 10 '16 at 23:53
34

It might be polite to write something like the following:

In Lemma 3.2.5, it seems that the authors assume that the ABC is XYZ, and that this is critical to most of what follows. But after studying the argument carefully, I don't see why this is necessarily the case. Could the authors please clarify?

In other words, give them the benefit of the doubt. If the authors are not cranks, then this will do as much good as claiming that they made a mistake, and will potentially cause fewer hard feelings. (And if they are cranks, you don't much care either way -- so may as well assume they're not!)

There is no need to continue with the referee report. Just send something like the above to the editor. The authors might withdraw the paper, or devise a workaround, or explain why their method is in fact correct. In any case, editors appreciate careful checking and you don't need to worry that such a brief message will make a bad impression.

11

The answer to your question depends on whether you think the mistake can be fixed with reasonable efforts (i. e. without redoing the work from scratch). This can be discipline specific, for redoing the analysis of a painstakingly assembled dataset is not the same thing as overhauling a purely theoretical work. If in your view it cannot be fixed, this provides ample grounds for rejecting the paper. Otherwise the referee should help the editor answer the crucial question whether to reject the paper or to request a major revision. In this case the rest of your review report should provide the clue whether the paper is worth revising and whether the authors are fit for this task.

6

I agree with Raghu's answer that you don't need to go through the rest of the paper until the error is addressed and with Anonymous's answer about how to phrase things. But if it were me I would also tell the editor something like (assuming this is what you think):

The results are interesting and important and would merit acceptance in this journal (after minor edits) if they are correct. So I'd recommend that the authors revise and resubmit if they are able to clarify the above point or replace this with a different argument.

Basically you want the editor to know whether they should be encouraging resubmission of a corrected version. You want to be clear that it passed the initial "are these results interesting enough" screen before you got to the harder "are they correct" screen.

4

Personally, you are the referee, and thus by definition, you are the expert the editors and publisher are relying upon to deal with this sort of thing. Go with your gut.

You should probably write out just what you wrote above bluntly. There seems to be a mistake that makes interpretation difficult until the matter is cleared up. If the journal asks you not to put any recommendations for publication to the author directly, but they ask you for your opinion on the matter, you should make it clear in the confidential section that you believe any revision would require additional review.

1

If I were in your place, I would write to the editor explaining that I have these doubts on the manuscript, which I cannot verify based on my knowledge/experience, and you let the editor decide based on his experience and the other reviewer(s).

At the same time, I would mark it for rejection, propose my theory and make it clear that if they can answer to your concerns adequately, the manuscript is worth resubmitting (and possibly publish).

  • If you don't have the knowledge/experience, you have no business marking it for rejection. And if you mark it for rejection you can't make clear that resubmission should be permitted. These are incompatible options. If it should be reconsidered if the problem can be fixed, you recommend 'major revision'. If it shouldn't be reconsidered regardless, you recommend 'rejection'. – cfr Oct 9 '16 at 0:19
  • I think it's not so straightforward in this case, as the reviewer is mostly certain that there is a major methodology mistake. I think that when someone is assigned as a reviewer, they have to inform the editor of all their concerns and the editor will decide further. There is no automatic procedure. An editor actually reads our comments and can decide whether our reasoning for rejection or major revision is valid. (also the editor knows who you are and what experience you have when he assigns the review. he wants to know your opinion) – BioGeo Oct 9 '16 at 6:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.