Questionable peer review practice in top-tier journal

Question

I am a tenured professor of engineering at a major research university. Recently, I was asked to review a manuscript for a top-tier multidisciplinary scientific journal. The authors were all grad student or early career faculty, except for one who is a National Academy member (but neither first author nor last author), a well-known person in the field, and a close collaborator with the other authors and the handling editor.

The manuscript was of reasonable quality and worth publishing in a specialized journal. However, it had a couple of serious flaws that, in my opinion, made it unacceptable for a broad, high-impact journal. I legitimately felt as though I could not recommend it for acceptance, knowing that its publication in this journal would give it - and the flawed approach on which it is based - undue credibility.

The paper underwent three rounds of review. The other reviewer accepted after two rounds. I recommended rejection each time. Then the editor accepted the manuscript without even soliciting a third reviewer.

I think it is pretty obvious that the manuscript was accepted based on the reputation of the National Academy member (and their relationship with the editor), and not based on peer review. If peer review actually mattered here, then a third reviewer would have been asked or the paper would have been rejected. The fact that this did not happen makes it seem, to me at least, as though the outcome was decided before the reviewers even saw the manuscript.

So this leads to my questions.

First: Is my complaint legitimate? Is it a fair assessment that this journal seems to be using a "good ole boys" system for evaluation rather than taking reviewer reports seriously?

Second: If so, what should be done? How can I raise this concern with the journal without compromising my anonymity? What protections could I expect against retaliation from the authors, editor, or both?

Answer 1

I can only speak from my own limited experience as an editor. Reviews are really just expert opinion. Just because a reviewer recommends rejection doesn't mean the editor has to act on it.

At some point, beyond a certain threshold, there is no "right" or "wrong". You can feel that there are significant issues with a paper. Someone else might disagree. It's up to the editor to decide what to do with that. Sure, inviting a third reviewer probably wouldn't hurt. But clearly at some point the editor decided that either A) your concerns were addressed or B) they were not actually as fatal as you believed. And so they didn't seek another review.

So while I suppose your complaint is legitimate in the sense that it is your opinion and no-one can tell you your opinion isn't legitimate, I wouldn't call it objectively fair. You really have no evidence that your report was ignored. There were 3 rounds of review - this is the opposite of not taking your reports seriously. And it is not obvious (to me at least) that the paper was accepted because of the reputation of an author when it should have been rejected.

Also, you're saying that maybe this would be better for a lower impact journal. But if the paper is as seriously flawed as you seem to think, it shouldn't be published anywhere. But if it isn't fatally flawed then we are right back to where we started - that it's okay for the editor to decide your concerns had been sufficiently addressed.

In any case, if you feel strongly that this paper slipped through the cracks, my only advice would be to find yourself a position as an editor so that you can make these decisions as you see fit. Going after the authors, journal, paper, or publisher at this point will not help anything. I suppose you could always write a letter to the editor detailing your concerns. These are not uncommon in my field, at least in the higher visibility journals. Though I would suspect that it would be tossed out since, essentially, an editor already decided that your concerns had been adequately addressed.

Answer 2

It's common practice in a lot of fields to publish rebuttals to other specific papers. E.g. "Comment on Spiderman et al.: Why two webs are better than one". The comment is also a peer reviewed paper in its own right, yet deals with highlighting perceived flaws in the original paper.

Science works best when discussions/progress happen out in the open so that the wider community learns what mistakes to avoid, or whether a widely-vaunted piece of work is actually flawed. Add your voice to the open conversation and publish your objections.

Answer 3

Journals publish what they want, and they can be pretty corrupt institutions. The problem for science is that your work on the article is wasted, because the paper was not corrected, and your objections are not publicly available. One remedy is to do peer review only for journals that publish the reports, in which case you can usually remain anonymous. Or publish your objections in another venue, such as a blog or a preprint archive. Professional scientists should not retaliate again professionally written criticism.

First: You have no legitimate complaint against the journal. Dispelling your illusions on the fairness and usefulness of peer review is not an actionable offense. Making you work for no result is also fair game.

Second: Seek another venue for publishing your objections. Decline to review for that journal in the future.

Answer 4

If we read through what you're saying the complaint isn't legitimate. The journal is obviously PNAS, so carefully read the title: Proceedings of the National Academy of Sciences.

It was founded as a place for the members of the NAS to report on their work. If you're not in the NAS, then you're secondary. In fact, prior to 1995, it was mandatory to have an author who had a NAS affiliation.

Is it a fair assessment that this journal seems to be using a "good ole boys" system for evaluation rather than taking reviewer reports seriously?

Yes, at least for consideration of novelty, and that's by design.

I would strongly suggest that you research and understand how this journal works before raising a fuss. Documentation is easily found: there is a special submission process for NAS members. Note specifically how novelty is evaluated by member-nominated reviewers.

Also observe how each published article will say whether it's "contributed" or a regular "direct submission".

How can I raise this concern with the journal without compromising my anonymity? What protections could I expect against retaliation from the authors, editor, or both?

You can't do anything. If you do, you'll look like a fool. You already may have by repeatedly bringing up novelty, which is more outside the independent reviewer's domain. If this wasn't a member-contributed article, another reviewer already looked at it for novelty.

They're completely transparent about this, so if you don't like it, then don't review for, submit to, or read PNAS. It's their journal, they can set the rules.

They're not the only ones either. For example, Cold Springs Harbor Journals are going to favor authors at Cold Springs Harbor Laboratory.

Answer 5

There is a lot to be said about the specific venue you are discussing (the topic is covered in this answer)

However, you are probably missing one important aspect of peer-review (as performed in many journals nowadays, please note I am not in favour or against, I just take it at the face value).

Reveiwers mmake suggestion to the editor, which then takes full responsibility of his/her/their decision. The reviewer do not decide anything. Even if three reviewers recommend rejection of a certain paper, it is still up to the editor to decide what to do.

The paper underwent three rounds of review. The other reviewer accepted after two rounds. I recommended rejection each time. Then the editor accepted the manuscript without even soliciting a third reviewer.

Probably your rejection was not strongly motivated. If you suggested rejection based on what you state in your question, if I were an editor I would be thankful for your work, but I would give a very light weight to your suggestions.

I would therefore proceed with doing a weighted average of the reviews:

• the two accepting it (maybe the other reviewer changed from one revision to the other, if you are 100% sure it is not the case please add this detail in your question);
• your review presenting no technical flaws, just some editorial (?) flaws that make it unacceptable for an ideal broad, high-impact journal.

And you know the outcome of this decision.

Answer 6

Is it a fair assessment that this journal seems to be using a "good ole boys" system for evaluation rather than taking reviewer reports seriously?

It is not usually provable in any particular case. However, I know from experience that it happens.

What should be done?

Stop hiring, promoting, and awarding grants based on low quality proxy measures. If people were rewarded for their research, rather than their publication list, then you would not be wasting your time deciding if a paper should be published in an elitist journal or in a specialized journal.

How can I raise this concern with the journal without compromising my anonymity?

The journal already knows who you are. You could share your concern with the editor-in-chief, but I would not expect that to benefit anyone. You would likely get ignored. Even in cases of blatant misconduct, journals are often very slow to react.

What protections could I expect against retaliation from the authors, editor, or both?

No "protections," but I would expect that nobody would have a desire or ability to retaliate. Maybe if they work at your university you might have a slight grounds to be concerned.

Answer 7

Embrace post-publication peer review. Write your issues on those questionable parts of the manuscript on pubpeer. People actually read it. And reputable authors respond there. This helps science evolve into more open discourse.