I wrote a thorough review that included many suggestions for improving the methodology, and criticisms of their analysis. Initially, the main problem with the paper was that the authors only reported significance tests (something like p-values) without reporting effect sizes. Since they provided their data, I calculated the effect sizes and found that they were tiny, and "significance" was only achieved through some dodgy methodological choices (artificially inflating the sample size). The second reviewer brought up some of the same concerns and the paper was rejected.
I was concerned about writing such a negative review, because I know it can hurt. But I did my best to write in a positive tone, and to make as many suggestions as criticisms.
I now see the paper has been posted as a pre-print, without any changes. If I were to see it posted in a journal, I would honestly say that it should be retracted because the conclusions are not supported by the data.
What do I do in this situation? I consider my options to be:
- Contact the authors and ask for a discussion. Perhaps some of my criticisms were wrong, and the authors did not have a chance to write a rebuttal since the paper was rejected.
- Wait for the paper to be published, and contact the editor.
- Post my review to the pre-print as a response.
- Forget about it, because life's short and who cares.
Update: Thanks for the enlightening discussion. I'll stick with 4 for now, since I don't know what the fate of the article will be (will it get published? will it be updated?). While nobody seems to suggest 1, I have a morbid curiosity about what kind of response I would get. This could always be followed up by 2 or 3. If I'm luck I'll have better things to do.