16

I wrote a thorough review that included many suggestions for improving the methodology, and criticisms of their analysis. Initially, the main problem with the paper was that the authors only reported significance tests (something like p-values) without reporting effect sizes. Since they provided their data, I calculated the effect sizes and found that they were tiny, and "significance" was only achieved through some dodgy methodological choices (artificially inflating the sample size). The second reviewer brought up some of the same concerns and the paper was rejected.

I was concerned about writing such a negative review, because I know it can hurt. But I did my best to write in a positive tone, and to make as many suggestions as criticisms.

I now see the paper has been posted as a pre-print, without any changes. If I were to see it posted in a journal, I would honestly say that it should be retracted because the conclusions are not supported by the data.

What do I do in this situation? I consider my options to be:

  1. Contact the authors and ask for a discussion. Perhaps some of my criticisms were wrong, and the authors did not have a chance to write a rebuttal since the paper was rejected.
  2. Wait for the paper to be published, and contact the editor.
  3. Post my review to the pre-print as a response.
  4. Forget about it, because life's short and who cares.

Update: Thanks for the enlightening discussion. I'll stick with 4 for now, since I don't know what the fate of the article will be (will it get published? will it be updated?). While nobody seems to suggest 1, I have a morbid curiosity about what kind of response I would get. This could always be followed up by 2 or 3. If I'm luck I'll have better things to do.

Improve this question
7
  • academia.stackexchange.com/questions/135326/… Related, might be duplicate.
    – Allure
    Commented Jan 30 at 3:49
  • 2
    Thanks, this is related. However there are enough differences between the questions that I would not say it's a duplicate. In my case the paper was rejected by the editor based on the reviews, and has now been posted as a preprint. Also, I'm considering a wider range of options as a response, including communicating with the authors.
    – jomimc
    Commented Jan 30 at 4:11
  • 2
    It is well possible that the next review process will request revisions from the authors before publication, and that the paper will be substantially improved when finally published in a journal. I'm just writing this because your question makes me wonder whether you are aware of that possibility. The community is normally aware that there is no quality control for preprints, so such issues in a preprint that is not otherwise published will arguably not do that much harm.
    – Christian Hennig
    Commented Jan 30 at 16:59
  • 1
    I'm not sure what's the norm in your field, but in my field anyone is free to post anything as a pre-print, in general, right? Even this includes unreviewed work. Maybe they wanted to post that pre-print before, but was held because it's still under review and the journal/conference doesn't allow posting a pre-print, so after it's done, they posted it as a pre-print, with the intent to update it with a revision later, etc. This is why general advice is to not cite pre-print, because it might be unreviewed, or incorrect, etc.
    – justhalf
    Commented Jan 31 at 8:25
  • 1
    A referee only makes suggestions for changes, the authors may or may not act upon these suggestions.
    – Tom
    Commented Feb 1 at 20:36

5 Answers 5

Reset to default
15

Think about how much of your own time you are willing to put into this. Assuming your conclusion about the effect sizes was correct, and the paper was methodologically dubious, I don't see what #1 would accomplish, since they have already gone ahead and published the preprint despite your reservations.

I have noticed that often people put up preprints (yes, even for somewhat incomplete, if not entirely dubious studies) to unofficially, but publicly, lay claim to a certain research problem — so they can later say, ours is the first result demonstrating effect X.

I can't claim to know their intention, but I would suggest skipping #3, give them the opportunity to revise and resubmit. Perhaps if you still feel strongly about it (if and) when it is published, you could go ahead with #2. Otherwise, there is always #4.

Just my two cents.

Improve this answer
3
  • 1
    Contact the editor to tell them what, exactly?? If the reference it to the journal that turned them down, they rejected the paper, and have no dog in this fight, at this point. If the reference is to the editor of the published paper, it's gone through their review process. What action would the new editor take? There's really no point to 2.
    – Scott Seidman
    Commented Jan 30 at 19:22
  • Scott, suppose that the authors really have no intention of making any changes to their work -- for whatever reason. The authors could try to publish someplace with a shoddy review process or the next reviewer entirely overlooks the issue OP found. In this event, while I am not specifically encouraging OP to take up arms and write to the editor of the journal with their concerns, I do think it would be nice if sometimes people did that. Notice that I say "Perhaps if you still feel strongly about it [...]". What the editor decides to do about this is a different matter.
    – principles-investigator
    Commented Jan 31 at 0:15
  • Again, this assumes that (a) OP has the time and capacity to do any of this; (b) they are confident about their analysis concerning the effect sizes; and (c) the authors of the study did not make revisions to their work.
    – principles-investigator
    Commented Jan 31 at 0:15
47

You have fulfilled your obligation as a reviewer ... but the authors whose work you reviewed are under no obligation either to agree with you, or to take any notice of your opinion. Indeed, it's always possible that, unknown to you, they received a very positive review from someone else!

Sure, if the topic interests you, and you want to post some kind of response, go ahead, but remember ... there are probably many more interesting things for you to spend your time on, including many more interesting things with which you have some disagreement and where your engagement will enrich your life. (As you remarked, life's short)

Improve this answer
2
  • I agree that the authors do not have to agree with me, and I am probably wrong on at least some points, however the other reviewer raised similar concerns and the paper was rejected. I can't imagine a competent reviewer failing to raise similar points (e.g. not reporting effect size). I genuinely think the work has such severe flaws that it should not be published anywhere in its current form.
    – jomimc
    Commented Feb 5 at 3:43
  • That said, I do have more interesting things to work on...
    – jomimc
    Commented Feb 5 at 3:44
8

1 and 3 are not options since they violate reviewer anonymity. You could ask the editor you reviewed for to contact them on their behalf, but I doubt they will want to get involved.

I would not worry too much for the moment. Consider the alternative scenario that they posted their preprint before receiving your review, and have not yet updated it. (Indeed, editorial workflows being what they are, can you be sure that isn't what happened?) Clearly there would be no reason to be concerned in that case. Is the actual sequence of events so different? After all, the effect so far is the same.

It is quite possible that they haven't had time to deal with the comments yet, but needed a preprint to be available immediately for whatever reason - claiming priority, one of the authors is applying for a job, etc.

Improve this answer
9
  • 14
    The reviewer is free to forfeit their anonymity. Or they can just contact them without mentioning the review (sure they will have their suspicions given the similar remarks).
    – Kvothe
    Commented Jan 30 at 19:10
  • 4
    @Kvothe no they are not. E.g. Elsevier's reviewer guidelines explicitly say "Please do not contact the author directly." Nothing special about that publisher, just the first one I looked at.
    – Especially Lime
    Commented Jan 30 at 20:45
  • 4
    @EspeciallyLime: It doesn't seem clear at all in which sense those "reviewer guidelines" can be considered binding (or what "binding" even means here). It seems more reasonable to argue with the culture of individual fields rather than with a publisher's generic reviewer guideline (which is somehow supposed to "fit" all fields and which most reviewer probably never read). From experience I can for instance say that many mathematicians disagree with your claim that must keep anonymity after the peer review process.
    – Jochen Glueck
    Commented Jan 31 at 0:43
  • A clarification: the preprint was posted several months after the review. My guess is that they have either given up on getting it published, or else it is about to be published in a journal. Time will tell.
    – jomimc
    Commented Jan 31 at 7:09
  • 2
    @EspeciallyLime: Editors of math journals are not different from other mathematicians; they share the same academic socialization. I heard an editor-in-chief of a well-respected math journal say the sentence "The referee is of course not obliged to stay anonymous". On another occassion the handling editor of a paper that I had reviewed asked me (not the other way round) for permission to reveal my identity to the authors, since the authors wanted to acknowledge me for the solution of a problem in their paper that I had included in the report.
    – Jochen Glueck
    Commented Jan 31 at 9:03
7

There are sites for public reviews, like https://pubpeer.com. You can post your reviews there.

Improve this answer
1
3

Post a comment. Posting of comments is explicitly supported on some pre-print servers, for instance on bioRxiv.

I think you are free to comment on the public pre-print just as anyone else would be. If you'd like to maintain plausible deniability that you were the reviewer, you could rewrite your review in different language and maybe focus only on one fatal flaw. The authors may still suspect you were the reviewer, but they would not know for sure.

You could also write a formal comment on the pre-print as a paper of your own and submit it to the pre-print server. These have been common in traditional journals for many years. For example, this ACS journal allows for "Comment" manuscripts that are "technical contributions that provide new information or an alternative perspective regarding a prior publication in the journal". I see no reason why this cannot be done for pre-prints as well, although it might require a significant investment of time on your part.

Improve this answer
1
  • I've done this with other papers, but not possible for this one unfortunately. Otherwise good suggestion.
    – jomimc
    Commented Jan 31 at 12:11

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .