Should workshop papers be cited (computer science)

Question

In my subfield of computer science I observe that workshop papers often are not thoroughly reviewed. It even appears (attention, possibly controversial assumption incoming) that workshop-paper acceptances are used to increase the number of participants at the conference, because authors of accepted workshop papers are required to register for the conference. Moreover, workshop organizers are often desparate to get enough submissions in order not to be forced to cancel the workshop. Hence, they are happy to accept submissions even if they do not completely fulfil certain quality standards.

Speaking about the quality of the papers, they often do not contain rigorous evaluations, hence, the presented results possibly are not really reliable (please note, that, of course, there are many terrific workshop papers out there!). I have a feeling that these papers would not get accepted at a (good) conference due to lack of maturity etc. I am aware that (especially in computer science) the publication path is often like this: early/initial work in progress is submitted to a workshop → if the work gets more mature it is sent to conferences → after some more work it is probably sent to a journal.

My question is: Given the above circumstances, should workshop papers be cited?

Since the quality (hence, the results) of such papers is often lacking, I ask myself whether it's a good idea to make a reference to it. One further thought is that references are often used to show that something has already been done or addressed. Therefore, the reference is used to argue that the problem addressed by the workshop paper should not be addressed anymore which is probably a wrong assumption if we assume that the work done in the workshop paper is sub-par.

Answer 1

Don't cite bad quality work as evidence. Judge the quality yourself. Only cite things if you think they are of a sufficient quality to demonstrate the claim you want to reference. Don't rely on peer review (whether in a journal or conference). Lots of poor quality work gets into journals, too.

Work in progress and early results do not equal bad quality. Something that is still in progress, or hasn't been rigorously demonstrated yet, may still be worth reporting. This doesn't make a bad quality paper, as long as the paper is honest about the interpretation of what is presented. Falsely claiming that early results are definitive is bad, but "Our early results seem to show this, but we aren't sure yet" can be a perfectly good paper. There is nothing wrong with citing such results, as long as you similarly are honest about what (and how much) they show.

In short, There is no reason for a blanket refusal to cite conference papers. Such a practice might make you miss important evidence. Do consider the source and the quality of the work, and try to cite the strongest source(s) when possible. This will often (but not always) mean preferring a journal article.

I'm assuming you are thinking of cases where you are searching for evidence in a particular area and can choose what to cite. Of course you also have an obligation to cite anything your work relies on or uses, regardless of the source.

Answer 2

dan1111's answer is good advice, and I thought I'd add a complementary perspective.

First, as a CS researcher, at least in systems or architecture, conferences are basically the top of the food chain. Journals are rarely more novel than the conference version. Usually a journal version has a specific reason: too much data to fit in the conference version and it's actually interesting, proofs/implementation that don't fit but are interesting, or (you'll like this one) a small piece of work that gets tacked on instead of being submitted to a workshop. I think your perspective on workshops is basically correct, except that---speaking as a previous workshop organizer---I don't care how many people attend the main conference at all. Organizers, at least in my experience, are usually just researchers who are trying to promote more discussion and interest on a specific topic they care about (and yes, we sometimes scramble for submissions so we don't have to cancel, but usually because we just want people interested and talking). But I'd like to add that there's also a number of decent workshop papers that exist because they're just not quite big enough to be competitive in a tier-1 conference. So instead of writing a rather dull, stretched 12-page paper, you write a neat, self-contained 6-page paper. Some of these can go on to get a decent number of citations, and deservedly so. So workshops can be more than preliminary work. You're probably well aware of this, but I thought I'd point it out.

Second, I think there's a somewhat more productive way of looking at related work: they're not your competition, they're your context. Let's say you've written X. It's great. It has novel contributions A, B, and C. Two years ago, someone published Y in a workshop. It's written kind of poorly, and it proposes a weaker version of A and B, but not C. Do you cite it? I think there are two strong reasons in favor: (1) they thought it was a good idea, too, so leverage them as motivation. Their paper exists as evidence that someone else thought the idea was interesting, too. If you've solved the problem in a better or more clever way, that is a great contribution which is not diminished by other people working on it, just like your work won't be diminished if someone builds a Z that extends your ideas later. (2) I feel it's far more common to get a negative review because you didn't cite a paper ("the authors fail to cite author et al., 2004. This is clearly the same idea") than becuase you did ("they mentioned author et al., but is this really different?"). If anything, citing a paper is a chance for you to choose the terms on which the paper will be discussed. I.E.- if you don't cite it, then a reader might think that your paper is different is some small way over in one area, but overlook bigger differences in another. If you do cite it, you can highlight exactly the differences you want. You can drive the discussion.

So I wouldn't worry about people thinking less of your work because you cite a weak paper. Obviously, don't spam citations into your paper, but take the opportunity to discuss, on your terms, why your work is so awesome.

Answer 3

Be open to cite any paper or source you consider to have had constructive value to your work. Sometimes reviewers aren't as good as the impact factor or acceptance rate might imply. (Acceptance rates are an artificial metric anyways, considering those conferences that extend the deadline until they have the critical amount of papers to pose with a low acceptance rate - only a side note).