10

I am a tenured professor of engineering at a major research university. Recently, I was asked to review a manuscript for a top-tier multidisciplinary scientific journal. The authors were all grad student or early career faculty, except for one who is a National Academy member (but neither first author nor last author), a well-known person in the field, and a close collaborator with the other authors and the handling editor.

The manuscript was of reasonable quality and worth publishing in a specialized journal. However, it had a couple of serious flaws that, in my opinion, made it unacceptable for a broad, high-impact journal. I legitimately felt as though I could not recommend it for acceptance, knowing that its publication in this journal would give it - and the flawed approach on which it is based - undue credibility.

The paper underwent three rounds of review. The other reviewer accepted after two rounds. I recommended rejection each time. Then the editor accepted the manuscript without even soliciting a third reviewer.

I think it is pretty obvious that the manuscript was accepted based on the reputation of the National Academy member (and their relationship with the editor), and not based on peer review. If peer review actually mattered here, then a third reviewer would have been asked or the paper would have been rejected. The fact that this did not happen makes it seem, to me at least, as though the outcome was decided before the reviewers even saw the manuscript.

So this leads to my questions.

First: Is my complaint legitimate? Is it a fair assessment that this journal seems to be using a "good ole boys" system for evaluation rather than taking reviewer reports seriously?

Second: If so, what should be done? How can I raise this concern with the journal without compromising my anonymity? What protections could I expect against retaliation from the authors, editor, or both?

14
  • 4
    academia.stackexchange.com/questions/135326/… Closely related, maybe duplicate.
    – Allure
    Nov 28, 2023 at 23:50
  • 30
    You are making a pretty strong assumption along with the accusation: "pretty obvious".
    – Buffy
    Nov 28, 2023 at 23:58
  • 13
    @mrp: you say “a couple of serious flaws that, in my opinion, made it unacceptable for a broad, high-impact journal”: why would the journal “impact” matter? Either the flaws are serious and should be corrected prior to publication, or they are not and then should not be a barrier to publication.
    – user126108
    Nov 29, 2023 at 1:37
  • 6
    "I think it is pretty obvious that the manuscript was accepted [..] not based on peer review." If so, then why did it undergo three rounds of review? Why not accept the paper after just one round?
    – JRN
    Nov 29, 2023 at 5:44
  • 5
    "The manuscript was of reasonable quality and worth publishing in a specialized journal" Enough said. The editor perceives its own journal as a publishing venue for specialized articels. It is up to them if 3 years down the line other journals will be " broad, high-impact" and their journal will just be a specialized venue... you do not have much to do with these strategic decisions.
    – EarlGrey
    Nov 29, 2023 at 12:04

7 Answers 7

40

I can only speak from my own limited experience as an editor. Reviews are really just expert opinion. Just because a reviewer recommends rejection doesn't mean the editor has to act on it.

At some point, beyond a certain threshold, there is no "right" or "wrong". You can feel that there are significant issues with a paper. Someone else might disagree. It's up to the editor to decide what to do with that. Sure, inviting a third reviewer probably wouldn't hurt. But clearly at some point the editor decided that either A) your concerns were addressed or B) they were not actually as fatal as you believed. And so they didn't seek another review.

So while I suppose your complaint is legitimate in the sense that it is your opinion and no-one can tell you your opinion isn't legitimate, I wouldn't call it objectively fair. You really have no evidence that your report was ignored. There were 3 rounds of review - this is the opposite of not taking your reports seriously. And it is not obvious (to me at least) that the paper was accepted because of the reputation of an author when it should have been rejected.

Also, you're saying that maybe this would be better for a lower impact journal. But if the paper is as seriously flawed as you seem to think, it shouldn't be published anywhere. But if it isn't fatally flawed then we are right back to where we started - that it's okay for the editor to decide your concerns had been sufficiently addressed.

In any case, if you feel strongly that this paper slipped through the cracks, my only advice would be to find yourself a position as an editor so that you can make these decisions as you see fit. Going after the authors, journal, paper, or publisher at this point will not help anything. I suppose you could always write a letter to the editor detailing your concerns. These are not uncommon in my field, at least in the higher visibility journals. Though I would suspect that it would be tossed out since, essentially, an editor already decided that your concerns had been adequately addressed.

10
  • 9
    @mrp The editor does not simply count votes, he/she reads the written opinions of the reviewers. So he/she might have just thought that the other reviewers had better arguments than you. Nov 29, 2023 at 22:44
  • 2
    @justhalf In my experience, it is a rare occasion that a journal editor shows any sign of intelligent life. It's worse than just counting votes. I'm sure plenty of journal editors will reply to this comment disagreeing with me. However, the journal editors on this site are not a representative sample of the ones I interact with, if you can count form letters as interaction. Nov 30, 2023 at 3:11
  • 1
    "But if the paper is as seriously flawed as you seem to think, it shouldn't be published anywhere." This misinterprets the question. Nov 30, 2023 at 3:12
  • 8
    @AnonymousPhysicist The question only makes sense in the first place if we assume that the paper is so fatally flawed that it could not have been accepted without nepotism/incompetence/whatever. But by the OP's own admission the paper would be okay for a lower impact journal. At that point, it sounds like a difference of opinion.
    – sErISaNo
    Nov 30, 2023 at 6:00
  • 1
    @sErISaNo You cannot know that without reading the journal's publication criteria. Nov 30, 2023 at 14:24
14

It's common practice in a lot of fields to publish rebuttals to other specific papers. E.g. "Comment on Spiderman et al.: Why two webs are better than one". The comment is also a peer reviewed paper in its own right, yet deals with highlighting perceived flaws in the original paper.

Science works best when discussions/progress happen out in the open so that the wider community learns what mistakes to avoid, or whether a widely-vaunted piece of work is actually flawed. Add your voice to the open conversation and publish your objections.

3
  • This does not address the issue in the question: That the paper was published in the wrong kind of journal. Nov 30, 2023 at 2:59
  • 2
    @AnonymousPhysicist what does it mean for it be to the "wrong kind of journal"? A specialized journal is specialized because they choose to accept specialized papers for publication, and vice versa for a broad journal. The journal's reputation is a reflection of the work that is published there, and not from decree of the Editor. If the editors start accepting specialized work to a purportedly broad journal, then perhaps the journal is not so broad after all.
    – Jeff
    Nov 30, 2023 at 17:39
  • @Jeff "The journal's reputation is a reflection of the work that is published there, and not from decree of the Editor." I would expect the editor or publisher of an elitest journal would disagree with you. They say the work published is good because they picked it. Nov 30, 2023 at 18:01
12

Journals publish what they want, and they can be pretty corrupt institutions. The problem for science is that your work on the article is wasted, because the paper was not corrected, and your objections are not publicly available. One remedy is to do peer review only for journals that publish the reports, in which case you can usually remain anonymous. Or publish your objections in another venue, such as a blog or a preprint archive. Professional scientists should not retaliate again professionally written criticism.

First: You have no legitimate complaint against the journal. Dispelling your illusions on the fairness and usefulness of peer review is not an actionable offense. Making you work for no result is also fair game.

Second: Seek another venue for publishing your objections. Decline to review for that journal in the future.

1
  • 1
    This. If, after raising concerns in a review, the journal decides those concerns are not important enough for them, it makes me reconsider whether the journal is good enough for me. By reviewing for a journal, I contribute to the quality and legitimacy of that journal. If I think that my contributions are not valued enough, I'll just stop contributing to it in any way (reviewing or submitting). Granted, this is more for when I consider that the editor ignored a fatal flaw I highlight, rather than serious, but still publishable.
    – penelope
    Nov 29, 2023 at 17:20
5

If we read through what you're saying the complaint isn't legitimate. The journal is obviously PNAS, so carefully read the title: Proceedings of the National Academy of Sciences.

It was founded as a place for the members of the NAS to report on their work. If you're not in the NAS, then you're secondary. In fact, prior to 1995, it was mandatory to have an author who had a NAS affiliation.

Is it a fair assessment that this journal seems to be using a "good ole boys" system for evaluation rather than taking reviewer reports seriously?

Yes, at least for consideration of novelty, and that's by design.

I would strongly suggest that you research and understand how this journal works before raising a fuss. Documentation is easily found: there is a special submission process for NAS members. Note specifically how novelty is evaluated by member-nominated reviewers.

Also observe how each published article will say whether it's "contributed" or a regular "direct submission".

How can I raise this concern with the journal without compromising my anonymity? What protections could I expect against retaliation from the authors, editor, or both?

You can't do anything. If you do, you'll look like a fool. You already may have by repeatedly bringing up novelty, which is more outside the independent reviewer's domain. If this wasn't a member-contributed article, another reviewer already looked at it for novelty.

They're completely transparent about this, so if you don't like it, then don't review for, submit to, or read PNAS. It's their journal, they can set the rules.

They're not the only ones either. For example, Cold Springs Harbor Journals are going to favor authors at Cold Springs Harbor Laboratory.

2

There is a lot to be said about the specific venue you are discussing (the topic is covered in this answer)

However, you are probably missing one important aspect of peer-review (as performed in many journals nowadays, please note I am not in favour or against, I just take it at the face value).

Reveiwers mmake suggestion to the editor, which then takes full responsibility of his/her/their decision. The reviewer do not decide anything. Even if three reviewers recommend rejection of a certain paper, it is still up to the editor to decide what to do.

The paper underwent three rounds of review. The other reviewer accepted after two rounds. I recommended rejection each time. Then the editor accepted the manuscript without even soliciting a third reviewer.

Probably your rejection was not strongly motivated. If you suggested rejection based on what you state in your question, if I were an editor I would be thankful for your work, but I would give a very light weight to your suggestions.

I would therefore proceed with doing a weighted average of the reviews:

  • the two accepting it (maybe the other reviewer changed from one revision to the other, if you are 100% sure it is not the case please add this detail in your question);
  • your review presenting no technical flaws, just some editorial (?) flaws that make it unacceptable for an ideal broad, high-impact journal.

And you know the outcome of this decision.

1

Is it a fair assessment that this journal seems to be using a "good ole boys" system for evaluation rather than taking reviewer reports seriously?

It is not usually provable in any particular case. However, I know from experience that it happens.

What should be done?

Stop hiring, promoting, and awarding grants based on low quality proxy measures. If people were rewarded for their research, rather than their publication list, then you would not be wasting your time deciding if a paper should be published in an elitist journal or in a specialized journal.

How can I raise this concern with the journal without compromising my anonymity?

The journal already knows who you are. You could share your concern with the editor-in-chief, but I would not expect that to benefit anyone. You would likely get ignored. Even in cases of blatant misconduct, journals are often very slow to react.

What protections could I expect against retaliation from the authors, editor, or both?

No "protections," but I would expect that nobody would have a desire or ability to retaliate. Maybe if they work at your university you might have a slight grounds to be concerned.

1
  • One problem with "stop using low-quality proxy measures" is that the alternative can be lower quality. The journal something gets published in is certainly not a precise measure of how good it is, but it is (or should be) based on expert opinions. My own (in most cases) non-expert evaluation therefore is likely to be more precise but less accurate. Dec 5, 2023 at 20:25
1

Embrace post-publication peer review. Write your issues on those questionable parts of the manuscript on pubpeer. People actually read it. And reputable authors respond there. This helps science evolve into more open discourse.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .