9

I'm mainly interested in the level of fact-checking of computer science conference papers. Do reviewers normally read the entire paper and exhaustively check the correctness of proofs, algorithms and claims?

I have read papers at top conferences where implementation details are sketchy and conceptual claims are inaccurate.

My guess is that an exhaustive level of fact-checking is often not done particularly in cases where the reviewer might not be familiar with all the concepts in the paper. Is this true? If this is so, then it seems unfair that the merit of a paper is essentially just based on how well the paper is pitched in the first one or two pages.

  • What area of CS is this? Papers about formal methods will differ vastly from computer architecture in terms of the things that can actually be fact-checked. – Zulan Mar 21 '16 at 14:49
6

I'm afraid there is no one-fits-all answer to your question. It highly depends on the tier of the conference. There are some that examine more into the work than others.

However, the level of scrutiny in CS conferences are generally less than that of a journal of comparable reputation. This is because, such journals go through multistage reviews and edits before the final print is published. Most conferences look only into the conceptual matter and usually have insufficient time to exhaustively implement and test the proofs of each of its candidate papers with limited reviewers.

Having said the above, there are some journals that scrutinize lesser than some conferences. Like I stated before, it all depends on the tier and reputation of the publisher.

  • 2
    As always, it depends on the reviewer. And better conferences tend to have better reviewers, or more conscientious ones, if only because they are in sight of line of reputable committees. – Captain Emacs Mar 21 '16 at 8:41
  • 2
    It's worth noting that conferences run on a tight schedule, whereas journals can take as long as they need to review a paper. This means conference reviewers often don't have time to properly check all of the papers they are given. – Thomas Mar 21 '16 at 13:25
  • 1
    It's also worth noting that conferences tend to have a strict page limit per paper. Many submissions are necessarily sketchy, just because there is not enough space for all the technical details. (Even if authors put the details into a technical report or an appendix, reviewers are usually not expected to check these in addition to the submitted paper.) – Uwe Mar 21 '16 at 13:59
0

I review papers in CS and I would say that it depends on the conference. If you submit to a very specialized conference, you have more chance that the reviewer will be very familiar with your topic. Thus he might check your paper carefully. But it always depend on the reviewer. Some reviewers will read a paper very quickly because they are busy (even for top CS conferences sometimes) or because they are less familiar with your topic. And some other reviewers will actually spent a good amount of time to understand your paper. So there is not general answer to this question. It depends on the conference and the reviewer. Besides, often a reviewer will first check your paper and if it is not good enough based on the introduction, conclusion and experiments or some other criteria, the reviewer may reject it without further checking the other sections. So this may also be a reason for not checking the details of you paper.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy