22

Suppose you were to find what you think are serious flaws in a preprint (arXiv or something similar). What should you do? Now, after the obvious first step of contacting the authors and letting them know, what should you do if you were basically told off ("thanks for your interest" without addressing any of the technical concerns)?

Were this a regular journal, one might write a letter to the editor, or a comment (one might go this route even without contacting the authors in the first place), but how should you deal with preprints, whose contents is in a way still evolving? Would a "comment" paper uploaded to the preprint server not be considered rather aggressive (I don't remember seeing anyone doing this)? Is there anything one can, or should, do before the (eventual) publication of the paper in some peer-reviewed venue?

Stretching this well into the hypothetical, suppose that one did write a "comment" paper and uploaded it onto the preprint server. Would the authors of the original preprint be ethically obliged to cite this and address the concerns raised therein when submitting to peer-reviewed journals (assuming that they were to submit after being let known of the existence of the comment)?

4
  • This is an interesting question. I did see a number of "Comment on..." on the arXiv, and instinctively it did strike me as aggressive, but I'm not sure that's the way everyone sees it. Some comments are definitely aggressive or even insulting, but others seem more civil so I'm not sure it would necessarily be taken the wrong way.
    – Ri49
    Commented Jun 3, 2014 at 19:01
  • 6
    You might consider how you would react if someone contacted you about one of your preprints. Extend the thought experiment from including your normal reaction to not reacting (because of family issues or other reasons that lower the priority of responding). How would you want that person to treat your situation? Commented Jun 3, 2014 at 19:19
  • What field? I think that matters quite a bit. Commented Jun 5, 2014 at 5:51
  • I have seen an example of this where the arXiv paper already has 10+ citations (but it's not published in a peer reviewed journal yet, after sitting on arXiv for a long time). To extend your question: is one morally obliged to do something about it and prevent others from (unknowingly) using flawed results?
    – Szabolcs
    Commented Aug 1, 2014 at 19:43

3 Answers 3

18

You have no need to do anything. By contacting the author, you pointed out what you believe to be a flaw. It could be you are incorrect, it could be that they deal with this in a future version, it could be that they simply don't care. To me, this situation is akin to the following xkcd comic:

enter image description here

People perform research, good and bad, every day. Once it's published you can feel free to write an article refuting their conclusions, but until then it's just informal academic discussion, and there's nothing you can really do about someone else being wrong.

Disclaimer: Some disciplines may treat preprints as actual publications, and if so, the preceding does not apply. I do not know of any fields for which this is the case, but then again, there's a lot of things I do not know.

3
  • 6
    I love your disclaimer.
    – CGCampbell
    Commented Jun 4, 2014 at 0:07
  • Theoretical Computer Science: we cite arXiv preprints more than often, we consider them to be correct in most cases.
    – yo'
    Commented Jun 4, 2014 at 22:26
  • I think that in many disciplines (such as those based largely on clinical trials) researchers simply accept that a lot of false results will be published. In others (math!) the attitude is quite different. That's not to say that math papers are never wrong (laugh); it's that people consider the publication of wrong papers to be a problem for the field. Commented Jun 5, 2014 at 5:50
8

As I understand it, the point of distributing (including posting, as on the arxiv) preprints is that one wants to both speed up and enhance the publicity of their paper and the feedback they get on it. A very common practice is to post preprints on the arxiv, see what kind of reaction you get, and then submit them for publication anywhere from a week to a few months afterwards.

So I think that it is fair to assume that if someone posts a preprint on an electronic server, they are interested in feedback from any interested party. Thus you were being helpful in contacting the authors about what you perceive to be a problem in their work. I think that in most cases, what the authors do with your feedback is their business, up until the point where it becomes submitted -- and you are an editor or a referee -- or published.

In particular, there is no "standard temporal scale" for modifying arxiv preprints: there are some famous examples of papers on the arxiv that have been known to be flawed for years but have not been "withdrawn". I would advise against that, but that's the authors' choice. At what point to make what you regard as less essential changes is really nebulous. To give one brief example, I found a lapse in the exposition of a paper I submitted to the arxiv last month just a few days after submitting it. I really wanted the paper to be understandable to everyone, so I did upload a new copy. More recently I discovered that a result that we attribute to mathematician X was also earlier done by mathematician Y (the two works were independent and use different methods). I certainly want to straighten this out before publication. But I have not yet uploaded a new version of the preprint: it's a judgment call.

Maybe the authors are addressing your comments. Maybe they happen to know that they're right and you're wrong. Or maybe neither of you is wrong and you're just not understanding the language in the same way. (I have seen this happen again and again.) It's hard for you to know what the authors are doing, and I wouldn't assume that your criticism hasn't gotten through: a bland acknowledgment of receipt is a pretty good first response to an email like that.

Should you upload a critical arxiv preprint yourself? Probably not, most of the time. Exceptions:

1) The mistake in the author's preprint is specifically detrimental to your own research. For instance if the information that the author's preprint is critically flawed is important to the refereeing of one of your papers, that's a reasonable call to action.

2) If the paper does something extremely important and is getting widespread "public" attention. For instance, if the preprint purports to solve the Riemann Hypothesis, everyone else believes it, but you have identified the flaw, then you'll be doing the community a great service by getting this information out.

These exceptions are fairly exceptional. Even in those cases I would counsel repeated contact with the authors rather than uploading the preprint straight away.

Important Note: I fear that my advice may be somewhat field specific. In mathematics, we really don't place a premium on openly critiquing each other: all other things being equal, that's a distinct negative. There are some academic fields where pointing out the flaw (or arguable flaw) in someone else's work is a really good paper of yours.

Finally, you ask if the authors would be ethically obliged to cite your hypothetical preprint. I think this is easy: you are ethically obliged to cite any source that you have specifically and substantively used in the course of doing your work or (especially) that has made you modify your work. So the authors would have to cite your preprint if they respond to it in any way. If their perspective is that your criticism is without merit, they do not need to cite it in their submission and it may in fact be inappropriate to do so.

7
  • Thank you for a comprehensive writeup! You say that in mathematics you don't place a high premium on critiquing each other openly. So are you saying then that you would not point out the flaws of a published paper, either? I am basically asking how much you would change your advice if instead of a preprint, I was talking about a published paper and asking whether or not to pursue a short comment-type publication? Would you suppose that a perceived critical flaw would constitute a reason, or would one "need" one of the "exceptional reasons" (as you put it)?
    – alarge
    Commented Jun 4, 2014 at 0:49
  • @amlrg: By that I mean that I would essentially never publish a paper which was primarily a critique of another paper. Pointing out the flaws is a different matter: I have corresponded with authors and editors about flaws in published work, which has resulted in the authors themselves printing an erratum (often mentioning me). Commented Jun 4, 2014 at 1:30
  • If you're curious about published papers, you should look at other questions on this site: it has been treated. Commented Jun 4, 2014 at 1:31
  • Thanks for the clarification. The published papers have indeed been treated elsewhere, but I felt that telling your opinion on those here would add to the information contained in your answer: that for you preprint/published does not change the situation much. Now I realize that we are coming from different fields, for your first thought was that the authors are probably thinking about my comments, whereas the thing that popped into my mind was that they might try to ram the paper through to a journal that will accept it as is. I most certainly hope you are right.
    – alarge
    Commented Jun 4, 2014 at 1:52
  • 2
    @tparker: I have seen your "What if..?" occur in practice. I think it's not great, but neither is it a definitively dishonorable practice on the authors' part: it is simply not part of the rules or norms of the arxiv that you must take down a paper that you know to be flawed by Time T, for any value of T. I presume that a positive proportion of papers on the arxiv are flawed, do not get published for that reason, and remain uncorrected. It makes you remember that the refereeing process -- which is slow and highly imperfect -- remains a necessary step. Commented May 25, 2018 at 23:35
1

There are plenty of papers on arxiv.org titled "Comment on... " e.g. http://arxiv.org/abs/0811.3876 (and also some titled "Response to ...") some addressing paper not yet published.

Regarding the last part of your question:pointing errors in a preprint is not rudeness. Ignoring such concerns is rude. A real life story is that of the flaws in two papers of Daniel Biss found by Nikolay Mnev (it seems that at first his private communications were ignored) and that forced Mnev to put a paper on the arxiv.

5
  • That there are comments on arXiv does not mean that the comments are directed at arXiv preprints. In your example, the comment was indeed written to address the problems of a paper that had already been published (although the arXiv title does refer to the arXiv title of the other paper, rather than the peer-reviewed publication). In any case, I have met people connected to the Ising debacle (your example), and it seems to be a rather long-standing feud and does not therefore set a good, general example.
    – alarge
    Commented Jun 3, 2014 at 18:31
  • I should also add, that I am not really interested in the highly specialized fields where arXiv preprints might as well be considered the main publication (say, theoretical high energy physics, or some branches of mathematics); In these fields, if arXiv is indeed essentially a journal of sorts, arXiv comments would perhaps not seem too out of place. Rather, I am interested in cases where preprints are essentially a way to disseminate the research to a larger audience, but where almost everyone will be citing the peer-reviewed version. I think this is the more typical usage of preprints.
    – alarge
    Commented Jun 3, 2014 at 18:40
  • I extended the answer. The Biss-Mnev story is told here: books.google.com/books?id=mXWr00HurmMC&pg=PA97.
    – adhalanay
    Commented Jun 3, 2014 at 19:34
  • I had not heard the story before, it was quite interesting. Thank you. It, however, is not related to what I am asking. My question is specifically about preprints (as opposed to peer-reviewed published material) and if it is appropriate to write public formal comments pointing out their problems. Were the preprint published (i.e. not a preprint), there would be clear venues (comment/letter to the editor), as the paper would have taken its final form. One should point out problems in published work, like your story suggests, but the situation is not as clear for preprints.
    – alarge
    Commented Jun 3, 2014 at 19:57
  • 1
    This doesn't really address the question at hand.
    – eykanal
    Commented Jun 3, 2014 at 20:03

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .