13

I'm reviewing a conference manuscript (computer science) and i found 2 other recent methods related to the same problem which were published on arXiv. So, i think it is highly recommended to compare their method to those ones. The arXiv papers also claimed to beat the state-of-the-art, but on different datasets, so one cannot compare them based on the reported numerical results.

So, should/could i say that it is a weak point that the authors have not included/used those recent relevant works in their introduction or evaluation basis? Or they do not have such an obligation as those papers have not been yet published in any proceedings?

Improve this question
5
  • 5
    You don't need to cite things you don't know about of course. The question really is whether you should know about them. This varies by field as noted by the answer here: academia.stackexchange.com/a/115771/75368
    – Buffy
    Commented Aug 24, 2018 at 11:15
  • Since notification tools (e.g., Google Scholar) can alert researchers to new results, they should know
    – user2768
    Commented Aug 24, 2018 at 14:28
  • 3
    As a stylistic note, it’s actually arXiv
    – Stella Biderman
    Commented Aug 24, 2018 at 20:53
  • 2
    @Buffy, "You don't need to cite things you don't know about of course." In general, I politely disagree. Publishing authors are responsible to become aware of what is currently published in their field. Looking at it another way, I would not consider, "I didn't know that article was published" to be an acceptable excuse for not citing relevant work. That said, my point doesn't necessarily apply to grey literature like arXiv: so, I certainly agree with you that whether or not someone is expected to know what is current on arXiv depends very much on their field.
    – Tripartio
    Commented Aug 24, 2018 at 21:55
  • @mikea, it would depend on the contents of the articles. If something newly appears on arXiv that convincingly contradicts the authors' findings, as a peer-reviewer I would at least clearly express my reservations to the authors and to the editor. Peer-review should respect scholarly accuracy more than a sense of that authors have a right to publish something just because they said it before anyone had time to contradict it. Or perhaps I misunderstand your implication?
    – Tripartio
    Commented Aug 25, 2018 at 6:16

4 Answers 4

Reset to default
17

I think this question is ultimately about standards within particular fields and societies. In math, where arxiv has been central for decades, it would be odd to overlook highly related examples, and using your language it's reasonable to assume authors would be obligated to check. In psychology, where I work, preprint repositories are still fairly unusual, and authors are not expected to review these sources.

Improve this answer
1
  • 3
    in math, where arxiv has been central for decades — ...or in theoretical computer science, where everyone uses Google to find references already. (See also: ECCC, technical reports, manuscripts on personal web pages)
    – JeffE
    Commented Aug 25, 2018 at 4:18
9

If you know about these results, then you should definitely mention them in your review. Since you are the sub-reviewer you can simply notify the PC members about this, and they will (or should) know how to treat ArXiV publications.

Usually, I think they will not consider it a detraction of the result. Also, they will need to assess whether the results have been obtained independently of the ArXiV papers or not.

Improve this answer
2
  • 1
    Usually, I think they will not consider it a detraction of the result. — That really depends on the age and content of the arXiv paper. If a paper solving the same problem as the submission has been posted on arXiv for more than a few weeks before the submission deadline, the submission is likely to be (and should be) rejected..
    – JeffE
    Commented Aug 25, 2018 at 4:21
  • I think few weeks is not that much. If it was like half a year maybe. I know of cases were this didn't detract the acceptance of papers; though this was for very prestigious journals, and it was claimed that the results were independently obtained, and so this case is slightly different.
    – Dilworth
    Commented Aug 25, 2018 at 12:59
8

Ultimately, scientific papers are supposed to further the "state of the art" in a field. For that, they need to compare with the state of the art, which includes any publication a paper's authors or readers can find -- whether it's in a peer reviewed journal or on a preprint server such as arXiv does not immediately matter.

So yes: you should definitely mention this!

Improve this answer
1

On one hand, I do consider it the job of reviewers to verify that authors are aware of relevant literature and to help them become aware of relevant work that they might have missed. So, I think you should definitely mention the arXiv articles that you found.

On the other hand, I think the authors should be given the right to decide whether they want to consider non-peer-reviewed work in their study, even if they are made aware of it. After all, they have submitted their work for your peer-review because they value the peer-review system; they should not be required to consider work as relevant that has not been formally evaluated by independent, qualified scholars.

So, in short, I think you should bring these works to their attention, but leave it to their discretion whether or not to incorporate their findings.

Improve this answer
4
  • ...because they value the peer-review system — [citation needed] It's at least as likely that they have submitted their work to the peer-review system because, to first order, only peer-reviewed publications count toward hiring and promotion decisions.
    – JeffE
    Commented Aug 25, 2018 at 4:23
  • @jeffe, yes: in your example, they value the peer-review system for its role in validating their publications for the sake of their career advancement--I very much have such scenarios in mind when I talk about "value". I did not necessarily mean that they value the system intrinsically.
    – Tripartio
    Commented Aug 25, 2018 at 6:08
  • @Tripartio : Quite many don't value the peer-review system particularly highly. It's more they need the peer-reviewed articles as it boosts their chances to be hired to spots they want in the future. Like a necessary evil.
    – mathreadler
    Commented Aug 25, 2018 at 11:15
  • Many researchers could be much more productive if they did not have to spend such large amounts of time and energy banging their head against the peer-review wall.
    – mathreadler
    Commented Aug 25, 2018 at 11:22

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .