If research design, data analysis, and results only are good, where is the line between suggesting major revision and rejection?

Question

Disclaimer: I am quite fresh as a researcher, even more as a reviewer.

I am reviewing a paper that appears to report a well-conducted empirical study, with proper data analysis. The results of the article are interesting and relevant for the journal.

However, I am very concerned with how the paper was written. I will provide some of my concerns of the report here:

Besides many grammatical errors - which I forgive as English is not my main language as well - the article is badly structured and often confusing. It fails to clearly state the research questions. No all the hypotheses are well supported by the literature review. It omits several key studies in the literature review. It often makes strong claims without a proper reference. It often fails to quantify stuff, falling to the tedious "a lot", "many", "few", "circa", etc. It is quite mysterious on certain aspects of the research design-e.g., a control group is suddenly mentioned without really clarifying what constituted the control group, and the paper reports that the experiment is not even a controlled experiment.

In sum, for the first sections I have nearly a comment for each of the manuscript paragraphs. I will do my best as a reviewer. I will provide all possible suggestions for improving the manuscript. Whereas I am confident that the results of the study are interesting, and that the research design is sound, I don't know if failing to clearly write the manuscript is a reason for suggesting rejection or for recommending major revisions.

Where is the line in this case? The journal does not clarify this in its reviewer guidelines.

In case of suggesting a major revision, what if the authors still fail to provide a clearly written and organized manuscript in a second round? Could I still request for major revisions?

Answer 1

1. I would recommend emailing the journal editor to ask for some of these clarifications. There might be certain unwritten policies endemic to that particular journal.

2. In general, you can ask for multiple rounds of revisions (major or minor) until you are satisfied. Peer review is supposed to be an iterative conversation. However, as I mentioned earlier, there could be certain journal policies regarding the number of revisions which you might get addressed if you talk to the editor.

3. As a reviewer, if you feel that the article should be rejected, you are free to make that claim. However, personally, I always give at least one chance to a paper (however horrible it is) to redress my comments.

Answer 2

Journal instructions vary so exactly what can be done may also vary. But, as a reviewer you re "only" suggesting a verdict based on your review. An editor will make the final decision based on your and one or more reviews.

One question you can ask is whether the revisions you suggest can easily be accommodated within some reasonable time frame. As an example, I edit a journal where we allow six weeks for "major revisions" (three for minor). Unfortunately, this time frame may not be obvious to anyone but the editor. But, you probably get a sense if the paper contains enough good science to be worth waiting for or if a new submission at some later time would be better.

Usually, you provide a report that is passed on to the authors in some way but you also provide a confidential note to the editor. I suggest that you detail your qualms about the paper and try to explain your thinking about the potential for this paper and whether it may benefit from either rejection and possibly later resubmission or by taking it down the "major revision" route. You can then provide you sense of how much time might be required for the changes so that the editor obtains a sure footing for the decisions. It is always good to receive a little of "pros and cons" from the reviewer.

In the end the editor will make the choice and it will be based on more views than just yours. Providing the best substantiated view you can will most likely be welcomed by the editor

Answer 3

It fails to clearly state the research questions. No all the hypotheses are well supported by the literature review. It omits several key studies in the literature review. It often makes strong claims without a proper reference. It often fails to quantify stuff, falling to the tedious "a lot", "many", "few", "circa", etc. It is quite mysterious on certain aspects of the research design-e.g., a control group is suddenly mentioned without really clarifying what constituted the control group, and the paper reports that the experiment is not even a controlled experiment.

I strongly recommend including this paragraph verbatim in your referee report.

This description is not obviously consistent with your claim that the paper "appears to report a well-conducted empirical study, with proper data analysis".  Is it actually possible, even in principle, to have a well-conducted study without a crisply stated research question or a clearly-defined control group, or a proper data analysis that uses "a lot" and "circa"?

These are very basic flaws, independent of the authors' first language. If they were spread throughout the paper, they would be sufficient to recommend rejection. If these problems appear only in the introduction (and later sections clarify the research questions, experimental design, and so on), then they may not warrant rejection, but they certainly require major revisions.

The discrepancy between the initial sections and the body of the paper suggests that a student wrote the former, and the PI wrote the latter. At least, I hope that's what happened. It's appropriate to be gentle (but firm) with the student, but a bit less forgiving of the PI for not advising the student better.

In case of suggesting a major revision, what if the authors still fail to provide a clearly written and organized manuscript in a second round? Could I still request for major revisions?

Yes.

If the authors' revision does not sufficiently address the concerns in your initial report — regardless of whether your concerns are about presentation or content — your response to the editor should be "The authors' revision does not sufficiently address the concerns in my initial report; therefore, I cannot recommend acceptance."

Answer 4

The difference between a major revision and a rejection can be thin, because after a rejection people can of course still significantly improve the article and resubmit. Many journals have a deadline to submit a revision. In this case, I think:

If the required work can reasonable be performed in the time scheduled for a revision, then recommend a major revision. If the work is so substantial that it would take a lot more time than that, then recommend rejection.

Answer 5

So, I'm in your place, reviewing the paper, and these are my (=your) findings:

The good:

• "A well-conducted empirical study"
• "Research design... [is] good"
• "proper data analysis"
• "The results ... are interesting"
• "The results ... are relevant for the journal"

The bad:

• "I am very concerned with how the paper was written." -> suggesting it could be written otherwise
• "grammatical errors" -> Rectifiable.
• "article is badly structured" -> suggesting a different structure would be good.
• "article is ... often confusing" -> given what you've written above, the confusing parts can be rewritten or removed.
• "It fails to clearly state the research questions." -> So it could state relevant questions; or none.
• "No all the hypotheses are well supported by the literature review." -> What, people can't make their own hypotheses? What you can really complain about is:
• "Omits ... key studies in the literature review." -> Solution is obvious.
• "makes strong claims without a proper reference" -> Claims can be weakened to fit the actual results.
• "Fails to quantify stuff [:] ... "a lot", "many", "few", "circa", etc." -> Rectifiable.
• "It is quite mysterious on certain aspects of the research design" -> Still, you decided it was well designed. Mystery could be dispelled, apparently.

Recommendation: Resubmit after major revision

Plain and simple. If the good part makes you want to publish the thing given a perfect writeup, then at worst they need to rewrite their paper. No "lines" here.