Immature papers on arXiv

Question

I am from computer science and we typically submit to conferences and less often to journals. Publishing pre-prints to arXiv becomes more and more popular in my field. From the discussion here on AS I get the impression that arXiv is more than just putting something on a personal website.

For example from What to do when you spot a paper on arXiv with the same essential material as yours? I get that I should discuss arXiv papers if they are related to your work.

Now, a paper for a CS conference contains often about 75% theory + 25% experimental section. The experimental section takes a lot of time - so I wonder (and fear) if there is a trend to upload just the bare minimum to arXiv in order to get credit for the idea.

Is this a problem? Maybe similar to patent-trolls, just uploading vague ideas in the hope to get citations?

Edit: For clarification: I am working in the field of data-mining, where it is common to have experiments to show that your idea does not only work in theory. This question/concern came up as we discussed the pro and cons of uploading our work to arXiv, since we have never done this before, but it seems to become more common in our field.

Edit 2: It seems that more people are concerned with this problem, especially in machine learning:

Yoav Goldberg:

This post is also an ideological action w.r.t arxiv publishing: while I agree that short publication cycles on arxiv can be better than the lengthy peer-review process we now have, there is also a rising trend of people using arxiv for flag-planting, and to circumvent the peer-review process. This is especially true for work coming from “strong” groups. Currently, there is practically no downside of posting your (often very preliminary, often incomplete) work to arxiv, only potential benefits.

Yoav Goldberg:

I do not mind posting papers quickly on arxiv. I recognize the obvious benefits of arxiv publishing and fast turnarounds. But one should also acknowledge its shortcomings. In particular, I am concerned about the conflation of science and PR that arxiv facilitates; the rich-get-richer effects and abuse of power; and some of the current arxiv publishing dynamics in the DL community. It is OK to post early on arxiv.

It is NOT OK to misrepresent and over-claim what you did. Sloppy papers with broad titles such as “Adversarial Generation of Natural Language” are harmful. It is exactly the difference between the patent system (which is overall a reasonable idea) and patent trolling (which is a harmful abuse).

[...]

Most people don’t read the papers in depth but only the title and sometimes the abstract and sometimes the intro. And when the papers come from established groups, people tend to trust the claims without verification. “Serious researchers” might not fall for this, but the general population sure does get mislead. And by the general population I mean people who are not actively working in this exact sub-field. This includes practitioners in industry, colleagues, prospective students, prospective reviewers of papers and grants. In the short time since this paper came out, I already heard, on several occasions, “oh, you are interested in generation? have you tried using GANs? I saw this recent paper in which they get cool results with adversarial learning for NLG”. This will be extremely harmful and annoying for NLG researchers who apply for grants in the coming year (remember, many grants are reviewed by a panel of capable but non-specialized experts), as they will have to either waste precious space and effort in dealing with this paper and with Hu et al and explaining why they are irrelevant, or be dismissed as working on this “already solved problem”, despite the fact that neither the paper in question nor Hu et al actually did very much, and despite the fact that both papers have terrible evaluations.

And the follow-up discussion on reddit

Answer 1

It seems to me that this question is less about the arxiv per se and more about how to navigate doing research in a very fast moving academic field.

I get the impression that arXiv is more than just putting something on a personal website.

It's certainly different. The main differences are:

(i) Many more people will see your paper.
(ii) Your paper will indeed be archived, essentially permanently. (Withdrawing a paper from the arxiv has the effect of uploading a new, empty version. Older versions are still there!) On your own website, you can take things down at least as quickly and easily as you can put them up.
(iii) Some (very obnoxious) journals may regard posting on the arxiv as "prior publication". (This is strictly unheard of in my field, mathematics. My guess is that CS is close enough to math so that it is at least very rare in yours.)
(iv) Minimum standards of completeness and professionalism are enforced on the arxiv. These are enumerated on the site itself, but the gist of it is that they are looking for manuscripts at the last step before conference/journal submission or later. They are not looking for early drafts.

Of these points, probably the last is most relevant to you. If it is standard in your subfield to include 25% experimental data [you say that is standard in "CS", but that is certainly not true across the entire field], then a paper uploaded to the arxiv without that would probably look to many in your field to be incomplete, which is against the spirit and perhaps the rules of the arxiv. So I wouldn't recommend it.

But the situation doesn't fundamentally change for papers that you or others post on your own website. The phrasing in your question suggests that you feel that you might not have to "be responsive" in the academic sense to papers that you find on people's webpages (only). That's not true. As an academic you have to be responsive to others' work wherever you find it.

In terms of the prospect of people uploading "the bare minimum to arXiv in order to get credit for the idea": is this an actual problem for you or just something you are wondering might be a problem? I have never encountered this problem in my work. That you are wondering whether it might be a problem makes me think you may be a quite new researcher and haven't fully grasped the way the academic community works. (Which is fine, and you have only to look forward to understanding it better. But you should talk to others, including advisors and mentors, to try to get a better idea.) Academia places a great privilege on completed work for exactly this reason. If you put out a manuscript which, say, modifies an algorithm and hints that it could be faster in some situations, the most likely reaction you'll get is "Go on..."

This question may finally have made me understand what people on this site are on about when they say things like "An idea is worthless". An idea is certainly not worthless, but a vague and unimplemented idea is of highly uncertain value, to the point where rushing to publish "only the vague idea" would be a very poor, um, idea.

By the way, you don't have to immediately drop something because someone else had "the same idea" and put out a paper before you. Much -- perhaps most -- important academic work overlaps with other work and even more of it refines and extends the ideas of others. How to respond to seeing "your idea" in another paper is a topic for a different answer.

Finally, let me say: if what you've done, are doing or want to do has real value, then it is unlikely to be received with thunderous applause this week and totally ignored next week. If you're living in fear that someone else will say what you want to say, maybe slow down and find more to say.

Answer 2

If there is a trend to upload just the bare minimum to arXiv in order to get credit for the idea.

No, there is not. No reputable author is stupid to publicly upload junk (on arxiv or anywhere else) and put his name on it. Perhaps, there are some people that do what you are suggesting (I cannot confirm though) but these mediocre-to-bad publications are nothing you should worry about.

Answer 3

Unfortunately, scientific communication is indeed in a state of treating it like IP for quite a while, otherwise known as "publish or perish". This has a few corollaries, most notable of which is that it all largely behaves like patents indeed.

However, if 5 people publish roughly the same idea, it is not a given the first of them would get all the credit and reference count that comes with it. If it goes really, really big - and few things do - yes, they would likely have their names up there somewhere. But for more mundane things like metrics the better connected and more visible ones win. That makes mature research still valuable and all that happens, really, is that people try to get more visibility for their ideas with publishing often and publishing lots. And arguably, that is a good thing.

Personally, I was against publishing immature work for many many years until coming across Richard Hamming's "You and Your Research". One of the takeaways was that the articles one writes are not monuments to their work - they might turn out to be so, and one may choose to present a body of work in its entirety, and there are still significant advantages to that... But ultimately, they are mainly communication. And that tends to speed up. For casting the greatest bit of the work in stone (or, in this case, paper) books exist.

Answer 4

I wonder (and fear) if there is a trend to upload just the bare minimum to arXiv in order to get credit for the idea.

You are letting the nasty worldly details of the race for academic credit, prestige, funds and jobs clound your judgement of academic actions.

If someone has a good idea, that has been developed enough to constitute 75% of a paper, but has not had the time or the resources to test it empirically - it is better for science that this person post some kind of note / mini-article / blog post about the idea for others to read.

Then, other interested people could collaborate with this individual who has perhaps resolved a problem they were tackled with, and now do not have to resolve themselves. Or even just undertake the experimental work themselves. This is also certain, by and large, to lead to some changes in details or perspective about the purely-theoretical part - as well as to ideas for future research.