-5

In which fraction of the recent papers is significant incompetence of review(s) observed in natural sciences? In my searching, I've only come across one example of such a review... is anyone familiar with any others?

  • 5
    While I'm not a highly published researcher, it seems to me that if a reviewer is that far off the mark with his/her/zirs/their comments, the paper may potentially need to be re-written to be clearer in the misunderstood sections. – tonysdg Nov 4 '15 at 14:13
  • 6
    This is currently a polling question, and needs to be revised to be a solicitation for answers rather than a poll. – jakebeal Nov 4 '15 at 14:13
  • 6
    @phys_chem_prof we're not here for discussion. We're here for Q&A. If your motivation is "I would like to have a discussion about ..." or "I would like to rant about ...", or "I would like people's opinions about ..." or "I'd like to take a survey about ..." then this really isn't the place for that at all. – EnergyNumbers Nov 4 '15 at 14:58
  • 2
    I think @tonysdg is on the right path. I never got a rejection for a paper. But, my coauthors and I made sure that they were clearly written to take the reader through the experiment, results, and discussion in such a way that there would be no unanswered questions. While it is easy to call the reviewers incompetent, and good writing is really hard (OK, really really really hard), perhaps you should look inward at least a bit. My take away from the listed comments that you objected to is that your paper was not clear enough, and that led the reviewer to go off the rails. – Jon Custer Nov 4 '15 at 15:08
  • 3
    @phys_chem_prof: but most questions on the website can qualify for a poll since an univocal answer does not exist to most of them – There is a huge difference between a (good subjective) question lacking a definite answer and a poll. In a poll I cannot agree or disagree with an answer (I cannot say whether the answerer gave an accurate account of their experience) – I can only post my own experience as an answer (thus we get tons of answers). In a good subjective question, I can agree with existing answers by upvoting. – Wrzlprmft Nov 4 '15 at 16:06
5

It's difficult to provide any sort of general assessment across all fields. My experience in peer-reviewed publication, however, has led me to conclude that truly incompetent peer reviewers are much rarer than it often "feels" to authors. It's always fun and cathartic to slag on Reviewer #3, but I disagree with your assessment of what makes an incompetent review.

Scientific communication is not a one-way street, although people often approach it in this manner. When you are writing an article, it needs to be well-matched in presentation to its audience. Comments of the form that you discuss typically reflect a disconnect between author and audience, resulting from one of two failure modes in the paper:

  • The material may not be presented as well as the author thinks. We get very familiar with our own work, and often start losing track of all of the assumptions and delicate connections between different aspects of the work. It's hard to linearize the complex tangle of relationships in most work into an accessible form.
  • The presentation may be good, but poorly matched to the audience. Different communities have different things they are looking for in papers and different contexts against which a manuscript will be judged. Often a paper that will be lauded by one community will be hated by another because the paper isn't appropriately presented for that community. This doesn't mean anything is wrong with the community, but rather with the authors' understanding of that community. For example, different aspects of some interdisciplinary work of mine have been published in both the programming languages and synthetic biology, and the two papers are utterly foreign to one another even though they are about essentially the same work with the same piece of software.

A truly incompetent review, on the other hand, is one that doesn't bother to particularly discuss the paper at all. For example, as a program chair a couple of years ago, I received the following review for one of the conference papers, as presented here in its entirety:

Nice work.

The reviewer was clearly happy with the paper, but this review gave essentially no meaningful input to the decision process.

So, after all of that preamble, let me return to the quantitative aspects of your questions:

  • My observation has been that the number of truly incompetent reviews is vanishingly small. Editors and conference chairs generally deal swiftly with reviewers who fail to live up to minimal standards.
  • Reviews showing a frustrating disconnect between reviewer and authors, like you present in your question, have been steadily decreasing over time as my skill as an author increases.
  • I spend very little of my time appealing or writing rebuttals---probably less than an hour per paper, on average. I spend much more time than that revising in response to reviewer comments, however, and my papers are generally much better for the work.
  • 2
    @phys_chem_prof I had a lot of experiences like you did as a graduate student. Nowadays, I still get them (less often, because I'm a better writer), but more importantly I now understand that a comment such as: "This phenomenon was already studied by $authorX" actually means "It is easy for researchers not familiar with your work to confuse it with the work by $authorX. You need to provide clearer explanation and evidence that differentiates it from that work." – jakebeal Nov 4 '15 at 16:51
  • 3
    @phys_chem_prof Please re-read my answer more carefully, as well as the one linked in "How to React to Incorrect Claims by Reviewers?" and you will find that my actual position is quite different than the caricature of it that you have presented. – jakebeal Nov 4 '15 at 17:12
4

Let's break the possibilities in two, shall we?

  1. The reviewers are indeed incompetent. If it is a journal, answer the remarks. If it is a conference, submit somewhere else. It happens. However, this possibility is rather unlikely and will add nothing to you as a researcher, so..

  2. Your paper is 'bad'. Don't be angry, but from your question, it seems like you failed to previously address the remarks. Most of the really good papers I've reviewed answered my potential remarks in the paper itself. For instance, they said that ""The applied method is wrong"", therefore, it would be good for you, and your paper, to have a paragraph somewhere saying "While it may appear that the method is wrong, we prove that it is indeed right because....". That way, not only you already cleared any ambiguities, you demonstrate that you reflected on the contents and possible ramifications of your work and that is important.

    You might have made some wrong assumptions about your audience, for instance, assuming that they will have a higher level than the reviewers actually have. This is a bad assumption, in general, because it is pretty rare to get reviewers/audience with a proper level. You really have to "take you audience by the hand" and guide them through the work, making the minimum amount of assumptions possible (leeway here if you don't have enough room).

Long story short, your paper let the reviewers misunderstand the contents, and that's bad.

Keep in mind that the writing itself is more important than the contents.... yes, I know, but it is true. It's easier to get accepted when you submit a very well written paper, on some not-that-good ideas, than a revolutionary idea, poorly written. Indeed, if the idea is that different (which is very good), the writing part has to be even better, to surpass the bias people have against new stuff and make them understand. (we usually try to understand stuff based on what we know, and new concepts that contradict it are harder to accept).

Of course, without properly reading the paper, this is mostly guesswork, but I've seen it enough to guess with some certainty. Nevertheless, if you paper is good, it will get better by doing this stuff, so no harm done.


That said, I'm not sure if I have n publications in the last y years and I don't care to count (my name is there, my scholar exists, have fun). Sorry, but you don't get to rant and pick who answers it, that's not how the world works.

I know you are probably fuming right now, because none of what I said is what you wanted to hear. Please, before you answer, calm down. I've been there, done that, it wasn't good for anyone. I'm on your side here, otherwise I could've just downvoted the question, which would be justified, from your tone alone and moved on. This is not the place for those angry rants.

  • 2
    You put this much better (and more nicely) than I did in my comment above. Perfect! – Jon Custer Nov 4 '15 at 16:02
  • There are no personal attacks in this answer - it is a blunt and correct answer. – user41783 Nov 4 '15 at 18:42
  • After the edits and comments removal, my answer doesn't make much sense... – Fábio Dias Nov 6 '15 at 15:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.