11

As a reviewer, what are some obvious signs to you that a paper is low-quality?

closed as too broad by Brian Tompsett - 汤莱恩, scaaahu, jakebeal, D.W., Wrzlprmft Jun 19 '16 at 7:46

Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer. Avoid asking multiple distinct questions at once. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.

  • 5
    A good paper needs to be three things: correct, relevant, and nontrivial. Insufficiency in any of these areas makes a paper low-quality. – user37208 Jun 18 '16 at 16:11
  • 11
    @user37208: fourth thing: readable. – Cliff AB Jun 18 '16 at 18:08
  • 1
    Fifth thing: novel – Karnivaurus Jun 18 '16 at 23:22
  • @Karnivaurus Wouldn't relevance imply novelty? – emory Jun 19 '16 at 0:21
  • @emory Well I suppose it depends on exactly what relevance means... To me, it just means that the paper contains work related to the field of the journal / conference, and related to the problem it is trying to address. For example, a paper which repeats some existing work could still be relevant, if the conclusions from that existing work are relevant themselves. Although I actually think that "relevant" is broad a term here -- because you could say that a paper is only relevant if it is correct, only relevant if it is non-trivial, and only relevant if it is readable.... – Karnivaurus Jun 19 '16 at 1:55
18

Kind of a broad question, but I'll play along.

It is easier to say what makes a paper high quality: I learn something new and interesting by reading it. A twinge of jealousy -- as in, "Gosh, I wish I had thought of / done this" -- is a good sign there. The absence of all this makes the quality of the paper not too high.

There are some papers that don't teach me anything new and interesting that I nevertheless feel are of medium (publishable) quality: sometimes I don't really understand the value of a result but can't claim (or can't justify a claim) that the results will not be of value to someone else or someone in the future. Sometimes the results are known to me in some form but don't appear in the literature. Sometimes the results are novel when viewed from the knowledge base and comfort zone of one subfield but less so from another (or perhaps from the vantage point of a true expert -- in a field where true expertise is rare), and these can be worth publishing.

Probably what remains is low quality. I am inclined to object a bit to the framing of the question in terms of "obvious signs": as a referee, I aspire to read a paper carefully enough to be confident in my evaluation of its quality. Obvious signs may inform my preliminary impression, but I will read further to confirm. But here are some things that would make me evaluate a paper as of low quality: as I go down the list, the quality decreases:

  • The results of the paper already exist in the literature. Usually this means the author was not aware of them.
  • The results of the paper already exist in the literature, in a better form than in the author's paper. Thus publishing the paper could actually be a step backwards.
  • Same as the above, but the current paper omits a key technique or perspective that is part of the contemporary understanding of the topic. For instance, there are some expository papers, and these do not need to contain new results or even the sharpest possible form of the old results it exposes, but the author needs to have at least the level of mastery of the topics being exposed as an advanced student in the area, or the exposition could have a negative effect.
  • The results are faulty, either in ways that are not easily corrigible, or are easily corrigible but when corrected yield absolutely nothing new.
  • The results look fishy, but the exposition is so obscure that it is a chore to tell right from wrong from basically-right-but-highly-garbled from totally absent.
  • The author cannot or will not make a clear distinction between what is attained in the paper and what is desired to be attained. For me and many others in my field (mathematics), this is certainly the worst. I saw an especially good example of this recently: a paper posted this month on the arxiv begins (immediately following an abstract which is at turns obscure and vacuous) by stating "Registration contains colored markers:" A red dot stands for "a fact which is not proven at present or an assumption". A yellow dot stands for "the statement which requires additional attention". A green dot stands for "statement which is proved earlier or clearly undestandable [sic]". And then immediately following that, still on page 1, there is a "THEOREM", which is a statement of the Riemann Hypothesis (for those who don't know, this is arguably the single most important unsolved problem in all of mathematics; inarguably solving it will get you $1 million)...marked with a red dot. Then we have "PROOF", followed by 28 more pages of colored-dotted mathematics, which I spent a minute flipping through out of sheer morbid curiosity. But if I were a referee then, rather exceptionally, this first page would have been all the sign I needed to be confident of the low quality of the paper.
  • 2
    +1 for mentioning the recording of folklore (readably and correctly) as publishable work. There is still too much opposition around to this simple idea. – darij grinberg Jun 18 '16 at 17:20
  • Based on your second paragraph, I have to ask: Do your favorite articles have proofs or methods that seem obvious only after you read them? – Forever Mozart Jun 19 '16 at 5:39
  • 1
    There's also the class of low quality papers that are so poorly written that it's hard to assess the other points listed. – beldaz Jun 20 '18 at 0:39
5

I tend to look for three things:

  1. Clarity : can I easily understand the paper? Are the language / logic / symbols coherent? Complex ideas can be hard to grok, but are worthwhile.
  2. Correctness : are there any mistakes in the presentation, mathematics or assumptions?
  3. Contribution : does this paper present something new?

If any of these three is lacking, then I usually mark the paper down (reject, or accept with major modifications).

Of these, the last (contribution) is the most important.

A low quality paper will miss on two of the three.

If a paper misses on 1 and 2, then I generally can't see whether 3 is true or not.

  • With regard to 1.: in my field the most important papers rarely pass your first criterion. Often they turn out to be understandable only after weeks or months of hard work on the part of the referee. But maybe you are talking about papers that are not as easy to understand as they should be because the exposition is poor? – Pete L. Clark Jun 18 '16 at 16:37
  • 2
    @PeteL.Clark yes, exposition not meat. Hard to understand because the ideas are complex is ok; hard to understand because the authors can't string two sentences together is not. – Peter K. Jun 18 '16 at 16:56
  • Okay, sounds good to me. – Pete L. Clark Jun 18 '16 at 17:21
  • Curiously, these are the same three criteria that I was considering for rating articles: researchpracticesandtools.blogspot.com/2014/03/… – Sylvain Ribault Sep 24 '18 at 7:53
  • @SylvainRibault ooooh! I like that. The A, B, C, D non-numeric ratings make much sense. Thanks for the pointer! – Peter K. Sep 24 '18 at 15:51
3

Mismatch between advertising and content

The unifying theme with low quality papers is a large difference between what they claim they've achieved and what they show they've achieved. Generally, in such a paper the abstract will look exactly like an abstract of a strong paper, but the advertised promise is not fulfilled by the paper content.

A quick and obvious way to look for that is the following - read the abstract to identify the most significant claims; then read the conclusions to see if the claims are still there (or the vague description in abstract implies something much more valuable than actually concluded), then skim the content chapters to see what evidence they provide that the conclusions are valid to their full extent. Good papers will contain exactly what it says in the description, poor papers will have done something small, but puffed it up to seem valuable.

  • Is it okay to present more than you advertise? For instance, what if you solve one really sexy open problem, which you advertise in the abstract, and then you use the same methods to solve another lesser problem. The answer seems to obviously be yes, but I worry about losing the reader or giving signs of obfuscation. – Forever Mozart Jun 19 '16 at 5:42
1

In addition to the important things mentione by Peter K. and Pete L. Clark, I would add the question "If you had these results and were not under any obligations to publish, would you invest the work of writing it up?". The answer is yes, if you think that the results are interesting in themselves, or might be useful to someone. The answer is no, if the reason to consider the problem is to write a paper.

The difference between these categories is not too easy to judge. Even if someone answers a question someone else asked, it may be that the question was only asked because asking lots of questions means lots of citations. In practice a slight feeling of disgust is sufficient to reject a paper, unless you are reviewing for a rather obscure journal.

1

I would like to add another aspect, that was not mentioned in the answers by Peter K. and Pete L. Clark.

Reproducability - If the description in the paper is not enough to reproduce the results and there is no supplementary material, then it could be a bad paper. This is especially regarding computer science and the presentation of novel algorithms. If there is not enough information for other research to implement it themselves and there is not reference implementation available, then I would wonder if the results presented are valid. Be aware that the author might only make the implementation available on request.

This might also be applicable to other fields.

If this happens to be the case and some other point from the list is true, I would consider it a bad paper.

1

I don't see mention that you can't (imho) have a high quality paper without a well researched citation list to the literature (I suppose there's a possibility that some concept has no relevant previous literature, but I can't think of any example of such a thing). Inadequate citations/references is a red flag. Vagueness, methodological sloppiness or inadequacies, lack of sufficient statistical power, are all also red flags. If the comment made above that some reviewers must spend months (multiple dozens of hours?) in order to review a paper is true, then there seems to me that quality must be discipline dependent. This brings up the question of the division of labor between the reviewers, editors, and readership. Who is responsible for determining "correctness"? In other words, where should "peer review" happen in the "scientific method"?

Not the answer you're looking for? Browse other questions tagged or ask your own question.