12

I am reviewing a paper 'X' that is heavily built on an existing paper 'Y'. The paper Y is available on arXiv; however, it has not been peer-reviewed yet. It is only 4 months since it has been available on arXiv.

As a reviewer, is that my responsibility to verify the claims made in paper Y? or should I simply assume that to be correct and on that basis review paper X?

4
  • Wouldn't Paper X validate Paper Y? As long as paper X validates and confirms it can recreate the results from Paper Y, doesn't that give some validation to Paper Y?
    – Issel
    Jul 15 at 20:43
  • 1
    @Issel Here, in some sense, paper 'Y' has been used as a black box to get paper 'X'.
    – IY2
    Jul 15 at 21:22
  • What does your question have to do with the literature-review tag that you used?
    – Tripartio
    Jul 16 at 8:18
  • 2
    Paper Y is the literature, that I am not sure if I should review. 🤔
    – IY2
    Jul 16 at 8:24
27

Given that refereeing is a voluntary and little rewarded task, I don't think it is fair to demand verifying more than the paper at hand. The report should make clear what the referee knows and what they don't know. E.g., it could read:

The paper under review contains interesting new results, and the proofs are (to the best of judgement) correct. They do, however, rely substantially on results from the preprint Y which I have not read in detail. Should Y contain a substantial mistake, this could easily invalidate most of the results in the paper under review.

The ball is then with the editor to make a choice. Wait for Y to be published? Go ahead? Gather more information?

0
7

Actually, it would be a bit dangerous to assume that a paper on arXiv is correct. If you want to give more than a pro forma review then you need to look at the earlier paper to at least get an idea of its correctness.

If anything stands out as problematic then it will certainly affect whether you can recommend acceptance of this paper.

Alternatively you can just reject the review and tell the editor that the earlier paper needs to be validated first, delaying a proper review of this one.


For a conference submission, which has a firm deadline, I suggest passing the responsibility back to the Program Chair. They can, perhaps, offer advice. But it is their responsibility to put the program together in a timely way. Just give them whatever information you can to help them do their job. Heroic efforts are not required of reviewers, but a PC might just have a hero reviewer in reserve for hard cases. It may be that the situation you describe just means that this paper isn't yet "ripe" enough for this conference. The PC may be conservative in their judgement to preserve the integrity of the conference or might go "all out" to get a proper (complete) review. But that is their responsibility.

0
-2

Good question. I provide two points of view:

a) According to the scientific community, anything not peer-reviewed is non-verified. Thus, you could reject the paper because it builds on non-verified sources. Building on non-verified (=non peer-reviewed) claims violates "the code of conduct" for research. In computer science, this "code of conduct" is not even known to everyone, but in other disciplines even conference papers are considered as non-properly reviewed and a poor source. The authors should have been aware of this. It is certainly not on you to review two papers, but, in practice, I often end up reading multiple papers for a review, e.g. to assess contributions of a paper or to verify claims of a paper.

b) Many high-quality papers get published on arxiv.org and they are also cited a lot. While arxiv.org does not do any real reviewing, they do a very lightweight check of publications (e.g. is it copy&paste from somewhere else...) and they ensure that only members of reputable institutions can publish there. The average quality of an arxiv.org paper is arguably at least the level of a lower quality conference or journal. Furthermore, many reviewers only conduct "coarse" sanity checks. My guess is that less than 50% of issues in technical details (proofs) are caught by reviews. So, building upon an arxiv.org paper is arguably as good as using other sources.

What is the recommendation?

If it is a high profile conference/journal and the paper's claims require the non-peer reviewed source to be correct, you should probably reject it.

If you want to play it safe, reject.

If the claims are non-essential and it is a really good paper, you might give it a chance, you could lower your rating because of this and also state the issue clearly in the review as others suggested.

4
  • 1
    Thanks! for the answer. I agree with the first point. For the second point, I was under the impression that anyone can upload his/her paper on arXiv. 😕
    – IY2
    Jul 15 at 17:48
  • 4
    "anyone can upload his/her paper on arXiv" is roughly true, and there is no assumption that what is there is correct. There are a few rules: arxiv.org/help/submit
    – Buffy
    Jul 15 at 19:20
  • 2
    In which fields "even conference papers are considered non-properly reviewed"?
    – P. Shark
    Jul 15 at 22:51
  • @P.Shark Being from the Computer Science field, I would say that almost all conferences impose a page limit of 9-12 pages for the main body of the paper. The content that is deferred to the appendix is optional for the reviewers to read. Therefore, it is natural to infer that conference papers are not properly reviewed.
    – IY2
    Jul 16 at 3:21
-4

As a reviewer, is that my responsibility to verify the claims made in paper Y? or should I simply assume that to be correct and on that basis review paper X?

Neither. It is your duty to verify the claims in paper X. Paper Y has not been peer-reviewed, and even if peer-reviewed you should know before hand if it holds true, no "assumption" here.

In theory and in an ideal world, if it has been peer reviewed it has been accepted by the community and it is part of the state-of-art knowledge on the topic[1].

Reference to paper Y are absolutely needed? if yes then the paper message is not adequately supported. If not, they should not appear in the paper, but they should belong in the acknowledgments (maybe the authors of X are trying to show they are not trying to copy work done by someone else, because they are just extending on the idea published in Y and they want to explicitly mention this).

[1] yes, the reviewer has such an extreme important role. Don't take it lightly.

6
  • 4
    You seem to be expected a level of self-containedness of articles that makes no sense at all in mathematics.
    – Arno
    Jul 15 at 12:59
  • @Arno either the content of paper Y is true, or it was true until confuted. No speculations, no assumptions, no self-containedness. Are we on SE:Mathematics, by the way?
    – EarlGrey
    Jul 15 at 13:08
  • 1
    Sorry but "either the content of paper Y is true, or it was true until confuted" isn't how math works. It is just the opposite. It is "unknown" until proven or disproven. Actually, the same is true in many fields. Anything can be claimed. Few can be shown to be true.
    – Buffy
    Jul 15 at 19:16
  • So @Buffy you mean that the conclusion "Chloroquine is effective in preventing the spread of SARS CoV in cell culture" from this paper ncbi.nlm.nih.gov/pmc/articles/PMC1232869 is unknown? I thought that it was true, and it was as true as what it says, that spread of SARS CoV in cell culture is prevented by Chloroquine. Then, if you build paper X citing Y, stating that Chloroquine prevents SARS CoV in human, then it is unknwon, until the conclusions of the paper, which must be supported by the paper content.
    – EarlGrey
    Jul 15 at 19:35
  • 2
    I'm not qualified to judge the cited paper, but if it hasn't been independently verified then nothing is "known", just claimed. However, if X "depends" on this paper (as opposed to just citing it), then X is questionable until the cited paper is validated. Such papers can have errors - even intentional ones, though, as I say, I can't judge the case. The cigarette companies "proved" pretty conclusively (???) that smoking doesn't cause cancer. Lots of citable literature.
    – Buffy
    Jul 15 at 19:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.