6

As a reviewer, should I point in the comments that, "I" think it is not right to cite a paper that is not peer reviewed -- arXiv papers, rather, cite a closely related/relevant peer reviewed paper. Do they expect me to pseudo-peer-review that arXiv paper too?

Edit: To be clear, I never meant to say "do not cite" as in to hide the source. I should have been more precise with the wording -- "replace the citation with a closely related peer reviewed paper/journal".

9
  • 55
    Not citing your sources, be they peer-reviewed or not, is plagiarism. Please do not suggest that authors plagiarise.
    – penelope
    Commented May 1 at 16:27
  • 11
    A paper can cite a book, a news article or a lecture. None of those are peer reviewed. The validity of the science may be doubt, but a citation is just "hey I go this from here", even if "here" is a drunk grandma's ramblings. Commented May 2 at 15:41
  • 3
    It looks like the title and body focus on two related but separate questions. Sadly, the answers have set the scope in stone now, but perhaps you want to ask a new, better focused question in the future. Commented May 2 at 15:52
  • 6
    A pretty popular machine learning method (rmsprop by Hinton) was never published as a paper. It was invented by Hinton during a coursera lecture. People just cite Hinton's coursera lecture slides. Commented May 2 at 19:56
  • 3
    @RicardoCruz Similarly for example for YOLOv3 -- that was only ever published on arXiv as a techical report. 27k+ cites to date. "Replacing with a closely related peer reviewed paper" would mean citing a different version -- so misleading.
    – penelope
    Commented May 3 at 9:46

4 Answers 4

57

No, I don't think this is appropriate.

Citation is about giving proper credit. If an author uses something from an arXiv submission to motivate a way they identify or approach a problem or the methodology of their paper, they must cite it. They don't get to choose to pretend like it doesn't exist merely because it was not peer-reviewed. It's not their sole responsibility to get that paper through review and published, so by insisting that any cited work be peer-reviewed work you're imposing an impossible burden on the authors.

If you have an issue with the specific claims made in reference to a preprint, you should make your comments about those specific claims. If it's sufficiently important to the manuscript, yeah, you might need to read another paper as a reviewer. That doesn't mean you must review every paper that is cited, though, you're reviewing the present manuscript not the whole literature.

Peer review is not magic. Lots of published, peer reviewed content is absolute junk. Peer review does not make something true. An author should put as much effort into understanding and evaluating any paper that they cite as if they themselves are a reviewer, whether it was previously peer reviewed by others or not. Similarly, a reviewer should evaluate all claims in a paper they are reviewing, and not just take the ones that have a peer-reviewed citation next to them as unimpeachable facts.

Remember that peer reviewers are not testifying that each statement in a paper is accurate and true; that's the author's job. A peer reviewer is looking at the methodology of the work and providing the editor comments about the appropriateness of the approaches taken, conclusions drawn, and connections made to existing literature.

13
  • 9
    I agree, and I might add that if a paper relies in a fundamental way on an arXiv preprint, and if the preprint looks like it'll never be published (e.g., the author died), then as a reviewer it is appropriate to ask for proofs of these results to be added, so that the statements they are relying on do make their way into the published literature (still with credit to the original discoverer / preprint). Commented May 1 at 14:16
  • 29
    @goldfinch I'm not sure what your field is, but I'd say common sentiment in physics is completely different from what you express here.
    – Anyon
    Commented May 1 at 14:28
  • 27
    @goldfinch I don't think peer review is as magical as you're making it seem. Lots of published, peer reviewed content is absolute junk. Peer review does not make something true. An author should put as much effort into understanding and evaluating any paper that they cite as if they themselves are a reviewer, whether it was previously peer reviewed by others or not.
    – Bryan Krause
    Commented May 1 at 14:33
  • 14
    @goldfinch Citing something not peer-reviewed is same as asking some LLM models to come up with text and tables. Why should anyone be trusting a not peer reviewed paper, when almost all the journals/conference have rigorous review process? I disagree. It's completely different. Anyway, as a researcher you are vouching for the correctness of anything you rely on, whether peer reviewed or not. But the fact is that science moves fast and if you only ever cite peer reviewed things you'd be years behind. Get used to relying on yourself to check correctness, not the peer review process. Commented May 1 at 15:02
  • 14
    @goldfinch Those issues all seem quite orthogonal to the one you've asked about; I also challenge you about whether it's possible for non-early stage researchers to read all the papers and techniques at the modern rate of publishing. See also: meta.stackexchange.com/questions/66377/what-is-the-xy-problem
    – Bryan Krause
    Commented May 1 at 15:58
8

As a reviewer, you should be evaluating the research presented, and it's presentation.

The body of work the authors cite is part of that. They need to be citing the "right" body of literature. Sometimes that includes non peer reviewed work. For example, conferences are fast-moving research by design, and some of the work cited might be stellar, but just not all the way through the peer-review cycle yet. That can take a much longer time than the fast-moving research presented at conferences can tolerate.

Some stuff can be in the margins, making astounding claims, sub-par methods, and may use questionable literature to support it. It might be fair to encourage the authors of such a document to be more careful to defend their methods and findings from a better literature. The more astounding the claims, the more solid the necessary defense of those claims needs to be.

6

I agree with the other comments. I would add that, even if already peer-reviewed, if the alleged result(s) from a cited source are heavily relied on in a manuscript you are asked to review, and you have not read this source paper before, it is also your duty to check that these result(s) are used in an appropriate fashion by the author(s) of the paper you are reviewing. This compels you to have at least a quick look at the cited source.

0

Arxiv papers should be cited. When I checked related work of papers and found arxiv papers that were missing, I even complained as a reviewer. But I approach them with more skepticism than papers from well-established/ranked conferences/journals. It is also ok to express this skepticism in a review, but I would not say that it is big concern as almost any good paper is first put on arxiv and the quality of the average arxiv paper is not much worse from a not too well-ranked peer-reviewed conference/journal.

As a reviewer I do mostly not "pseudo"-review references because they are arxiv papers, but rather I look at references if I feel the need to.

However, sometimes I have stumbled upon papers that are 2-3 years old and still only on arxiv.org and some additional search revealed that they got rejected at good conferences for serious flaws (openreview.net). So here I have sometimes acted as a "meta"-reviewer.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .