5

Is there any good way to determine when it appropriate to reject a manuscript versus merely recommending major revisions? I.e., at what point do you declare a manuscript a lost cause? I understand there are some situations where the decision is very clear. For example, if a paper has methodology problems but is otherwise solid "major revisions" is an easy decision, as is the case if the manuscript presents clear examples of plagiarism or results that do not advance the field.

However, there are cases where the proper decision to make for a manuscript is not clear. As an example, I was recently asked to review a manuscript on this topic. Originally, I thought the paper was a straightforward "major revisions", but the more I look at the more I am not sure. Specifically...

  • The paper is poorly written and has a large number of grammatical and syntax errors. The journal is an English journal but I know authors are ESL speakers (they are from Argentina and France). In general, I have a strict policy as a reviewer not to reject papers for spelling/grammar/wording issues if English is likely not their first language since the issues could be out of unfamiliarity, but merely make suggestions as to how the syntax could be improved. So this is a minor issue from my standpoint but combined with the other issues it starts to add up.
  • The data are incorrectly formatted and input into their analyzing program of choice, which is likely causing bias in the data (and at the very least raise questions as to its results). Specifically, the data's formatting contradicts what was described in the methods. The methods state the values were based on the published values for the data points but the actual data corresponds to broader (and much less precise) categorical bins used in my field. They authors claim the data was compiled from the previously published literature but it is clear they merely got it from an online database, as it contains errors that are only in that particular database's version of the data.
  • Their definition of terms contradicts that of all previous studies, and they do not justify this change. They find a non-significant result but this is almost certainly because their data are nonsense because of noise introduced by their different definition of the term
  • Inconsistencies in reporting of the data. I.e., in some places the authors say two variables are negatively correlated and in others they are positively correlated.
  • The authors make claims in the discussion that they say are "evident" but provide no support for this either in their own results or via citation of the previously published literature
  • The authors do not sufficiently cite the previously published literature. Specifically, the authors appear to be avoiding citing papers from my research group. In their manuscript when reviewing the previous literature and in the discussion they bring up some of the exact same results obtained by previous papers by myself and my lab group, but do not cite them once. Checking through the references they cite pretty much every major research group working on this topic except ours. Discussions with other colleagues in the field suggest this may not be an honest oversight, but may be personally influenced. My concern is that leaving a highly critical review would result in academic retaliation or being blackballed by the authors, and our field is small enough it would be hard to leave an anonymous review.
  • The lead author is a graduate student (approximately my age), but the others are my senior by at least 10 years (I am a PhD student) I suspect the paper was primarily written by the graduate student based on literature errors that suggest a lack of familiarity with the subject. People have told me that if reviewing a manuscript from someone that is my senior I should be very lenient because they have power over me. There's is also the whole aspect of not being too hard on a very young researcher.
  • The paper is also scooping research that at least two other groups, one by myself and other colleagues and the other a third party, presented at the main conference for our field last year. The abstracts of these presentations were publicly accessible. I have no problems with this fact, I want to see this group's paper accepted. However, examining the data in detail suggests that they may have just thrown the dataset together using previously published databases without critical review. EDIT: Based on the comment by @Buffy I thought I should add a bit more clarification. Our research group was actually planning to scrap our project because we weren't making much progress on the manuscript and it wasn't high priority, and this will likely be the nail in the coffin. The issue is more the fact that our lab group presented on this same topic is publicly accessible knowledge.

The paper might be salvageable and contains results that would be worthwhile if they stand up to scrutiny after the data has been fixed, but at the same time it shows extreme sloppiness that would not be acceptable for any scientific publication. I suppose, more broadly the question could be formulated as if there is any point where comparatively minor issues are so numerous that rejection becomes a better option than major revisions, or is it always better to choose major revisions over rejection? As you can see, this is a pretty complicated situation that makes it difficult to determine whether major revisions or outright rejection is the better course. Personally, I'd like to recommend major revisions for this manuscript, but I'm not sure if it's the right thing to do. I usually don't like to reject a paper unless there is something fundamentally wrong with it that cannot be fixed with revisions.

I've talked about this with some colleagues, and I've had some who say I should not only not reject the paper, but I should also not bring up the issues as to improper data formulation, definition, and citation issues and give a recommendation of minor revisions at worst so as to avoid getting blackballed by the other researchers. But I'm not sure if this is the right choice.

9
  • 5
    This is a really long question, and the specific problems are a bit of a distraction. You should either ask about whether you should major revise / reject one issue or ask a more general question. Commented Jan 5, 2021 at 21:19
  • Maybe start changing into a clear brook of reasoning that feeds into this cesspool of cognitive biases, academic supremacy, and egotism? Out of all the comments you may have, divide them into "these are bad because they are objectively bad," "these are bad because I don't think they like me," and "these are bad because I don't like them," and submit the first category only? Sometimes it's that simple, speak issues that can be backed and can be addressed or improved. Life is too short, to-do list is too long for endless psycho-analyses. Commented Jan 5, 2021 at 21:26
  • 4
    @Penguin_Knight I don't get what you're getting at with this. The issues with the manuscript are methodological. I want to see their paper accepted. My concern is that they will interpret this as a personal attack and there will be negative repercussions for my career. The fact remains that they are a senior researcher and I am a PhD student and senior researchers generally do not like PhD students criticizing them. I only brought up the research politics aspects because it clarifies why I am concerned. Commented Jan 5, 2021 at 21:30
  • 3
    @user2352714 is peer review not blind in your field/this journal?
    – fqq
    Commented Jan 6, 2021 at 0:23
  • 1
    @fqq Not really. Additionally, the field is so small it's really hard to remain anonymous. If someone in this field were to submit a paper there are only about 25 people who would be asked to review it, and its really easy to narrow down who does based on writing quirks, fields of interest, etc. In the broader field it's often said in practice peer review is never blind because everyone is so specialized you almost immediately know who is going to be reviewing your manuscript. Commented Jan 6, 2021 at 6:35

6 Answers 6

16

Many of the issues you raise are not relevant. Who wrote it, for example. Likewise scooping isn't relevant, though you may need to withdraw due to a conflict of interest. If the research is really parallel, not plagiarized, then scooping is fine.

But for the main question, ask yourself, what would be the quality and impact of the paper if the errors are (mostly) corrected? Is it innovative? Are the main results important? Is it "interesting". If the answer is no, then the paper should be rejected. Otherwise suggest major edits and let the authors see what they can do.

But I think the conflict of interest issue is especially important in your case. If it is eventually rejected and it is learned that you were a reviewer but later published something similar it won't help your career. You will be open to a plagiarism charge whether warranted or not.

5
  • 1
    I agree the scooping isn't relevant, my concern is that the authors will interpret it as personally driven because of that when my concerns with the manuscript are methodological, especially since it would be difficult to hide my identity. As I said, I want to give a recommendation of major revisions with the intent of seeing those authors eventually getting their manuscript published. But I know other researchers might reject it because they have a low tolerance for a lack of sloppiness when submitting. Commented Jan 5, 2021 at 21:27
  • Right answer about the content. Be sure to raise the conflict of interest question with the editor. Then they can do what they wish with your comments. Commented Jan 5, 2021 at 21:27
  • 5
    If the other reviewers suggest reject than that is up to them. It is also something you should ignore.
    – Buffy
    Commented Jan 5, 2021 at 21:32
  • @Buffy Sorry, I meant more in the sense of "a different person in my position might choose reject, is that the right thing to do and am I making the wrong decision"? Commented Jan 5, 2021 at 21:43
  • 2
    Again, ignore what others might do. But it is a judgement call. Different people will call it differently. For your amusement
    – Buffy
    Commented Jan 5, 2021 at 21:46
11

If you think the issues with the manuscript can be fixed, the authors should be given the chance to do a revision - if those are major issues, or a large number of issues, then a major revision.

If you think the issues cannot be fixed (or after fixing it would be in essence a new manuscript), then the manuscript should be rejected.

6

If there were an objective way to answer this, peer review would not be so random. Unfortunately there isn't. Still, the standard guideline is that if the issues are fixable, recommend revision; if the paper is fatally flawed, recommend rejection. Alternatively, if the issues are (relatively easily) fixable, recommend revision; if they will take a lot of effort, recommend rejection.

Examples of issues to recommend revision:

  • Poor (but still comprehensible) English.
  • Paper does not cite X, Y, Z, but it should.
  • Paper does not clearly define [term] even though it uses [term] extensively.
  • Paper does not perform [cheap experiment] to verify [claim], but [claim] is plausible - recommend they do the experiment instead.
  • Paper has a curious feature in the data that could refute the paper's thesis, and the authors do not treat it - recommend they examine the feature instead.

Examples of issues to recommend rejection:

  • Incomprehensible English.
  • Paper is provably wrong.
  • Paper is provably inferior, e.g. a paper proposes a new method to do X with 80% effectiveness, but another known method is already capable of doing X with 85% effectiveness.
  • Paper is plagiarized / violates academic ethics (e.g. it is submitted without the consent of all authors).
  • The paper, even if correct, is not interesting enough for the journal (or not within its scope).
  • Paper's central thesis is [claim], which is not supported by the analysis conducted. For example, if the paper says people prefer X to Y, but the sample of people it used is clearly biased, and there is no analysis of how the sampling bias might affect the results.
  • Paper lacks basic features, e.g. it analyzes the effect of X on Y, but doesn't have a control sample even though they really should have designed the experiment with one.

All that said, not everyone will agree with all of the above, e.g. some might recommend the authors repeat an experiment with the control. So you'll have to exercise judgement.

Based on your description of the paper you are reviewing, it reads to me like it's more reject than revise. The most problematic points are points #2, #3, #4 and the last one. A paper that contradicts itself is a clear problem that should never have arrived at you in the first place (assuming you didn't misunderstand). Scooping is not so much a problem as it is the fact that the data was thrown together without critical review. If as you describe it the data is nonsense, then the results are in jeopardy and could be completely wrong.

If you are still concerned, you could recommend revision and give the authors a chance to improve their work. If the revised manuscript is still terrible, then recommend rejection.

3

None of these reasons are reasons to outright reject a paper. Those that are relevant are possible to fix.

  • The paper is poorly written and has a large number of grammatical and syntax errors.
    This can be fixed by copy editing.
  • The data are incorrectly formatted and input into their analyzing program of choice, which is likely causing bias in the data
    If the data are genuninely incorrectly formatted, rather than just not formatted to your taste, then, again you can ask them to fix this and rerun the analysis.
  • Their definition of terms contradicts that of all previous studies, and they do not justify this change.
    I'm not sure I understand how the definition of a term can add noise to an analysis, but irrespective, ask them to change there definition and rerun or justify their change.
  • Inconsistencies in reporting of the data.
    Ask them to clear up the inconsistancy. If they say positively correlated in one place, but negatively in another, ask them which it is.
  • The authors make claims in the discussion that they say are "evident" but provide no support for this either in their own results or via citation of the previously published literature
    Ask them to provide support for this claim or claims.
  • The authors do not sufficiently cite the previously published literature.
    Ask them to cite the missing papers.
  • The lead author is a graduate student (approximately my age), but the others are my senior by at least 10 years
    Irrelevant.
  • The paper is also scooping research that at least two other groups
    If you are worried about scooping then this is not relevant. If you are saying that they are plagerising, then that is relevant, and is the only thing here that is a reasonable reason to reject. But you should have solid proof its intentional plagiarism.

In general I only reject papers that have problems that are not fixable. All of the problems you have raised with this paper are fixable. Now the authors may decide they can't be bothered with all the issues you've raised, or the editor might decide that this amount of revision cannot be accomplished within the journals revision period, or they might fix the problems and their results, evaporate, but that's not something you can tell right now.

However, you say this:

and give a recommendation of minor revisions at worst so as to avoid getting blackballed by the other researchers.

If the paper is flawed, you cannot return a review saying it is not. But you do have the option of withdrawing from the review process if you do not feel able to give an honest assessment.

2

Age, and scooping are NOT reasons to reject a paper. Poor grammar and syntax, incoherent data reporting, and "that's evident" are.

Multiple uses of "It's trivial to show" or "It's evident that" were enough to make me reject a paper on those grounds alone (references to proofs should be included). Combine that with poor English and poor proof-reading it's likely the right call to reject the paper.

You cannot evaluate the contributions, importance, or correctness of the paper if proofs are left as an exercise to the reader.

You cannot evaluate the correctness if there are contradictory statements, such as claiming negative correlation and positive correlation.

Poor English skills alone shouldn't disqualify a paper, but if there are several it should be corrected before approval.

People have told me that if reviewing a manuscript from someone that is my senior I should be very lenient because they have power over me.

This is journal dependant, but most journals anonymized peer-reviewer feedback for this reason. You should give an honest assessment of the paper. Are you reviewing papers from your university (you should not do this)?

If you feel like minor corrections would fix this paper, then you can do a "reject with revision (aka soft reject)." It sounds like there would need to be significant changes, like added proofs. Be sure to write up these issues in the rejection reason. Point them towards existing work, and explain why the notations they used were confusing.

The reject review should stick to issues with the paper, like missing proofs and poor editing and English. Do not bring up age. Only cite the "scooping" if the results were already available to the community at large (otherwise this isn't an issue).

2
  • 1
    For poor but comprehensible grammar, you should recommend revision, not rejection. Commented Jan 7, 2021 at 2:51
  • @AnonymousPhysicist - If you re-read the poor English sentence, I said poor English skills alone shouldn't disqualify a paper. If the paper already has several issues (which it sounds like this one does), then it should be pointed out so the author(s) know to improve their English writing skill, and because it makes a paper with several other issues even harder to understand, and thus evaulate. Commented Jan 8, 2021 at 15:27
1

Is there any good way to determine when it appropriate to reject a manuscript versus merely recommending major revisions?

You list a number of arbitrary specifics—those are irrelevant. When making this decision, do not look at what the problems are with the paper. Look at what it would take to correct them, and be fair to the authors.

Put yourself in the authors' shoes. Would you be able to make the changes you think are necessary (is it at all possible)? Would you be willing to? Would it take a reasonably small amount of time compared to what the original work took?

If yes, do give them the chance to fix the work, and make it very clear what exactly needs to be fixed.

If not, recommend rejection.

What you should avoid doing is repeatedly asking for major revisions when you believe that they won't happen (to a reasonable standard) anyway. That creates false expectations and just wastes everyone's time. Suggest rejection—there's always the possibility of resubmission.

Not the answer you're looking for? Browse other questions tagged .