26

My theoretical computer-science paper that I submitted to an Australian (i.e., located in Australia this year) B-level (according to http://portal.core.edu.au/conf-ranks) DBLP-listed computer-science conference was accepted 4 weeks later, but the reviews are short:

  1. 5 sentences (accept)

  2. 2 sentences (accept)

  3. 3 sentences (reject)

  4. 3 introductory sentences + 3 sentences noting weak points + 3 sentences noting strong points (borderline)

All four reviews are superficial. They partially contradict each other. The buzzwords and phrases that occur there are: well organized, good presentation, minor format errors, sound results, no theorem proofs, highly relevant, well written, convincing, weak motivation, no empirical study, no clear justification of the contribution, only theoretical analysis, no explanation of contribution, no clear structure, results useful for practicioners and theoreticians.

There is nothing beyond these high-level claims in the reviews: no page numbers, no section numbers, no quotations, no citations, no related work, no examples/counterexamples. None of the reviews substantiates any of its claims.

It's the first time I get this kind of reviews. Usually, I get longer and deeper reviews that show that the reveiwers understood the technical contents of the submission at least a bit beyond the abstract and that elaborate on a technical level at least a little bit.

My questions are:

  • What is the likely cause of such short and useless reviews?

  • Is the conference really serious? (The paper submission site stayed open long after the formal submission deadline.)

  • Are my concerns groundless? Is what happened normal for B-level Australian conferences?

  • Is there any way to extract more information, say, by writing the PC chair(s)? If so, how do I formulate the message, or even what do I ask? Or is it better to keep silent and be satisfied with acceptance?

The proceedings will be handled by IEEE Conference Publishing Services.

I know that the exact answers might be hard to tell, so I'd be happy with answers based on well-informed guesses.

|improve this question
34

The basic idea is simple: reviewers have no incentive to write good reviews apart from their desire to see the conference succeed. Seeing an abundance of short, superficial reviews with a high acceptance rate indicates a low quality conference.

An alternative explanation might be that this is a non-archival conference, often referred to as a workshop. In that case reviewers mostly care about the general problem the paper presents and whether it sounds interesting/relevant. Program chairs are usually those who go through the papers to see what fits (and there’s a small number of submissions).

If it’s not a workshop, the program committee/organizers aren’t well-respected researchers, and they charge you a lot of money in registration fees, I’d consider retracting.

|improve this answer
  • Agree in general except that the sentence "reviewers have no incentive to write good reviews apart from their desire to see the conference succeed." is false. Sub-reviewers at least have no incentive to make the conference succeed. Why would they? – Dilworth Sep 3 '19 at 18:43
  • Sub-reviewers are not reviewers. They are often assigned by a reviewer (or program committee members as they're called sometimes), and do a good job because they're beholden to the reviewer (reviewer is advisor/mentor/friend etc.). Reviewers/PC members are selected and can choose to say no (I say no to conferences I don't enough care about all the time), and are almost never paid. – Spark Sep 4 '19 at 3:48
  • @Spark, "Sub-reviewers are not reviewers. "--- They certainly are in the context of this question. OP has concerns about the reviews he got. It is highly reasonable that the reviews were written by sub-reviewers not PC members. – Dilworth Sep 4 '19 at 15:37
7

My field is statistics and I know that in computer science conference publications have a different (higher) standing than in statistics, so what I write may only be marginally relevant. However, I would distinguish between the high level conferences with top-level high impact publications and high rejection rates, and conferences that are in the first place there in order to foster exchange and scientific discussion, where results are also published but not aiming at particularly high impact. In statistics this is the norm rather than the exception. It doesn't mean at all that the conference is "not serious" - it may still be a fine conference. However it won't give your paper a high impact. If that's not what you're after (and I have no idea whether your paper is of a standard that would allow you to aim high), you could well be fine with that conference (of course I can't know for sure not knowing the conference). I don't believe, by the way, that any conference will have negative impact on your CV. Zero is the worst that can happen, unless you boast about that conference as if it was the pinnacle of anyone's career.

As for explanations, it may be that they use a small pool of reviewers whom they send several papers with very tight deadlines. Then that's what you get. Not top level practice but not necessarily a sign for a crappy conference either.

|improve this answer
3

The fact that the conference is on an international rotation makes it at least decent. The issue here is the quality of the feedback and the frustration that the feedback has not provided more direction in your work. Unfortunately this happens. It could be the reviewer of your particular section. It could be that the conference organisers discouraged or made decent feedback difficult (as others have remarked here).

Maybe focus on getting more feedback and support at the conference itself? Write to presenters and people attending about your work. Organise a time to meet them. Attend relevant preconference workshops. These skills are important in developing collaborations and opportunities even if you are going to leave academia. Hopefully they would be happy to help and give your issues some thoughts and suggestions.

Slamming the conference and conference organisers is poor form. It could be that the conference is run by and managed by emeritus professors and this is their retirement gig. It could be that it is not as rigorous as other places. But you have to make do with the situation as it is rather than what you would like it to be. Plan to exit and platform out of your toxic environment as quickly as you can.

|improve this answer
2

What is the likely cause of such short and useless reviews?

This depends on the field I guess, but:

  1. People being reviewers in multiple venues with large numbers of submissions - not enough time to do proper reviews.
  2. People not taking their duty as a reviewer seriously enough (which also relates to...)
  3. Lack of oversight/meta-review by PCs / PC chairs / editors of reviewers' reviews.
  4. Reviewers not having the necessary background actually review the work.
  5. A scathing argument for rejection which renders a detailed review of the rest of the work irrelevant (at least in the reviewer's view) <- although that should not be useless
|improve this answer
1

I agree mostly with Spark's answer, but let me give my own bits:

  • The conference is not a good one. A B-level conference (based on CORE) means a not a very good conference. A B-level journal is not that bad, but for a conference the situation is different. That is, it is futile to have papers in B-rated conferences. Remember also that ratings start with A*, A and only then B. So a B-level means third-rate!

Conclusion: There is no reason in my view to submit to a conference which is not A* or A-rated since:

  • You get meaningless reviews like you did.

  • It does not contribute to your reputation (looks bad on CV).

  • It doesn't contribute to your research (in most cases you won't meet anyone to cooperate with in a B-rated conference, unless they are also doing uninteresting research).

  • It is better to submit to a good journal.

|improve this answer