8

Different conferences have different standards of accepting papers. A paper rejected at a conference A might be best paper at a conference B (in a lower rank).

I'm reviewing a paper for a conference, that is not really related to my narrow field, that I don't know its ranking, that I haven't read any paper from previous series.

The paper is so so, not too strong, not too weak. It would be a straight reject in some top-tier conferences, but might be accepted/weak accepted at some other lower rank conferences. How do I make a decision about this paper for this particular conference?

An obvious solutions is to read a couple of papers from previous series to have a sense, but given it is not very related to my narrow field, it will take me a lot of time, and I'm having several other deadlines.


UPDATE: to answer Pete L. Clark's comments.

So I think your first task must be to decide whether to referee the paper at all

I have never submitted to a journal before, but I think one of the main difference between journals and conferences is that: in a conference, the chairs can't always select the right reviewers for a paper.

In a conference, there is a fixed set of PC members. After the deadline for abstractions, the PC members start bidding for the papers they want/don't want to review. The PC chairs then have a painful task: assigns papers to PC members (there are tool supported, but mostly useless). Obviously, there is never the case that everybody is satisfied, it's very rare that you get the papers that you want to review, and it's not uncommon that you get the papers that you don't want to review. When you are a PC member, you can't refuse to review the papers assigned to you.

If the conference adopts double-blinded review, you are not allowed to assign external reviewers either. In the case you can ask external reviewers to do reviewing for you, since those external reviewers do the reviewing without any credit (at least PC members have their names in the conference's web site), in most of the cases you can only assign to your postdocs/PhD students. So now you understand why I have to review this paper, and that refusing to review papers is not an option (my boss is the unlucky PC member).

  • 2
    As a reviewer, is this your job? Isn't there an editor or program committee member making the final accept/reject decision? – sessej Mar 23 '17 at 16:43
  • 2
    In CS, all reviewers make their own decisions, then the PC members make the final decision based on those decisions. For example, I had a paper which all 3 reviewers gave "weak accepted", but then it was rejected. I had another paper which got both "accepted" and "weak rejected", and the final decision was "accepted". – qsp Mar 23 '17 at 16:47
  • 3
    Given you are not PC member, it's not for you to decide. You can recommend. If you cannot recommend, because you cannot judge the value of the paper, just be honest. – PsySp Mar 23 '17 at 17:22
  • 1
    @qsp: Thanks for the update; it definitely places the situation far outside of my experience. I really don't understand why it's okay for someone to assign their subordinates to referee a paper but not suggest a qualified outsider reviewer. "So now you understand why I have to review this paper, and that refusing to review papers is not an option (my boss is the unlucky PC member)." That makes me understand why your "boss" (thesis advisor? postdoctoral supervisor?) cannot refuse, because he agreed to the situation. It doesn't make me understand why you can't refuse... – Pete L. Clark Mar 24 '17 at 17:50
  • 1
    ...except generally that you have to do what your boss tells you. (To be honest, though I am a lifelong academic I find this remarkable: I can't think of a single situation in which I've compelled a subordinate to do something they didn't want to.) But it sounds like it would be most helpful to assume the premise that you have to referee the paper. So...okay, skip to the part of my answer where it is assumed you will be refereeing the paper. – Pete L. Clark Mar 24 '17 at 17:52
5

First let me say that I work in a field (pure math) that is closely related intellectually but for which approximately zero percent of papers are published in conferences. In this case I don't see how it matters that it is a conference rather than a journal except possibly to reduce the timeline until the decision (though some journals in my field are starting to ask for reports within a few weeks!), so I will answer based on my own experience.

The main problem here is that your expert opinion has been called for on something that you do not feel like a fully fledged expert and that (as is most often the case!) it is not practical for you to fix this by substantially increasing your expertise.

So I think your first task must be to decide whether to referee the paper at all. If you for instance you didn't understand the paper at all then certainly you should not be a referee. If you've already agreed to do it, that makes backing out of it more socially awkward, but it could still be the right decision, as it may take some reading of the paper to find the stuff you don't understand. If you vaguely understand it but really not enough to evaluate it in any meaningful way, again I think you should not referee it. If you like, you can explicitly offer to referee something else instead, and I'll bet the chances are good that a different paper could be found.

If you feel that you can or must referee the paper but really feel shaky about evaluating it, then you can write a report in such a way that your evaluation will be minimized, e.g. by explicitly writing something like

I was asked to review this paper, and so I will. But I want to be clear that it lies outside of my core area of expertise, so my recommendations about its suitability for the conference are tentative. I hope that more weight will be put on the other recommendations.

And then you can give a "weak" recommendation, which should then be drowned out by the others.

However, your question suggests that you are sufficiently qualified to referee the paper that the first alternative is not appropriate and the second one may not be either: you write

The paper is so so, not too strong, not too weak. It would be a straight reject in some top-tier conferences, but might be accepted/weak accepted at some other lower rank conferences. How do I make a decision about this paper for this particular conference?

[Point of order: a paper can't be "weak accepted" by a conference. It must either be accepted or rejected. So here you are doing a bit of what other people have brought up: conflating the referee's recommendation with the editor's decision.]

Rather importantly, you don't say why the paper would be a clear reject at top conferences. (Not the assertion is surprising: presumably the vast majority of submitted papers in your field fall into this category.) Knowing that means that you have some information and insight about the paper (and more than you've told us). In fact it might be helpful for you to tell us how you came to this conclusion.

Probably I don't need to tell you that if the paper does not contain any correct, novel work that is of interest to someone, it should not be published anywhere. Conversely, if it does meet these requirements it should be published somewhere. If you don't know enough about the standards of this particular conference, you can work around that by explaining you would recommend rejection in venue A but recommend acceptance in venue B (this is rather common information in referee reports, at least in my field). Then the editor can decide where the conference lies with respect to the data points you've provided.

If you feel qualified enough, then at a certain point you do have to impose whatever standard you feel is most reasonable. If you don't know the field very well and the paper is not interesting to you, then [assuming you've decided to go ahead with the referee job] you should recommend it for rejection: what else? On the other hand, if you find the paper to be at least somewhat interesting you should act so as to leave the door open for the paper to be accepted, in particular by writing that you would recommend it for acceptance at journal or conference X. Ultimately you're leaving a decision to the editorial board that they would have anyway: among papers that are publishable in absolute terms, do they want to publish these papers or those papers? If they get a whole bunch of reports of the form "The paper is okay; it could be published somewhere in between heaven and hell; I don't really have strong feelings about it" then they're going to have a problem....but who is to blame for the problem? Them, of course: they did not find the right referees and were not clear enough in their standards.

Let me end by saying that I have found myself in similar situations more than once: namely, I get asked to referee a paper by a journal I've never heard of, in a subfield of mathematics different from any of the ones I've thought deeply about. And in fact I have usually done more or less the above: sometimes I turn it down (and I have learned to be decisive about this; if I gave some choice to the editor, it always turns out that they want me to do it anyway), but if I can do it I often take the job. A paper should certainly look novel to the relative outsider if it will look so to the expert, and in some cases I have rejected the paper for not making clear progress over (even) its own citations. More often the papers have been a bit interesting. Sometimes I have found significant mistakes: I can't remember a situation where the mistake was so bad that I outright recommended rejection, but there have been situations in which revisions have been necessary in order for the principal results to look correct. (I don't know how this plays out for the shorter timeframe of a conference: presumably outright rejection becomes more likely.) In fact the most common outcome is that I understand the paper well enough and think it's somewhat interesting and novel, though certainly not the kind of breakthrough to be published in a higher tier journal. In these cases I have recommended the paper for acceptance and included in my referee report an honest depiction of the situation: e.g. if I am unfamiliar with the journal, I say so. I believe in every such case the paper has been accepted. This has been fine for me, since having been an author many times and an editor never, fundamentally I am more sympathetic to the situation of a solid paper being rejected than to the plight of a journal that publishes a good paper rather than a great one. "A tie goes to the author," I feel. Given the number of papers I've refereed, this attitude seems to be okay with the editors, who do in fact once in a (great) while cheerfully reverse my recommendations.

2

If the paper isn't related to your field and you're not familiar with the conference, how did you get roped into reviewing it?

But given that you did, I suggest you provide detailed feedback on the paper, without focusing too much on the actual recommendation. As a commenter mentioned, it doesn't sound like you're a PC member, more an external reviewer, so if you either leave it to the program committee to evaluate based on your feedback, or offer a weak reject or weak accept rating (if you have to) with text privately to the PC saying that you aren't calibrated for the conference, that should be enough.

I do not think you need to spend inordinate amounts of time reading other papers from the conference simply to gauge its competitiveness.

0

First, some conferences will have two (or more) reviewers per abstract/paper, so the conference planners might consider consensus. Your review alone may not be the deciding factor.

Second, the goal of conference presentations is completely different that that of journals. They are meant for the discussion of and generation of new ideas. So, the stakes are fairly low if you have a paper that is on the fence (especially if you cannot evaluate the paper within its specific field). If you can give qualitative feedback, that would be helpful for the conference committee. However, when I review conference papers I always consider that the author may be a graduate student or junior faculty. This is a learning experience for them. So, my advice? If the topic is relevant and would be interesting to the conference audience there is fairly little harm in accepting. Every conference - even prestigious ones - have some poor presentations (either poor topics or poor presentation content/delivery). If the one you accept ends up being not that great, the conference's reputation is not necessarily harmed.

  • "Second, the goal of conference presentations is completely different that that of journals. They are meant for the discussion of and generation of new ideas. So, the stakes are fairly low if you have a paper that is on the fence (especially if you cannot evaluate the paper within its specific field)." I think that is true in some fields and really not in others. E.g. in some branches of computer science, conference publications are viewed as more prestigious than journals and therefore most academics don't publish in journals anymore. – Pete L. Clark Mar 25 '17 at 5:33
  • " However, when I review conference papers I always consider that the author may be a graduate student or junior faculty. This is a learning experience for them." That's interesting. In my field (pure mathematics) there are almost no conference papers, and when I review journal papers I always look up whether the author is a graduate student or junior faculty member. – Pete L. Clark Mar 25 '17 at 5:34
  • Thanks for the information about your discipline, Pete! I will keep that in mind in future responses. In terms of your second comment, the conferences I have reviewed for in the past have been blinded, so that may help or hurt some of those who submit. – Nicole Ruggiano Mar 26 '17 at 20:46
0

Why should the ranking of the conference matter? Presumably your decision should and will be based on contents of the submission, not on the prestige of the event.

Select events will attract a larger number of "good" papers, and so you might have to reject a larger number of submissions but otherwise let the science decide for you. Good is good and bad is bad in any context.

  • To be honest, I read your answer as "Why should the quality of the conference matter? It shouldn't influence your decision. Except if the conference has high quality, in which case it may influence your decision." Kind of puzzling. Do you mean perhaps mean that a paper that is truly "bad" should not be recommended for acceptance by any conference? (If so: sure, I said that too.) – Pete L. Clark Mar 25 '17 at 5:40
-1

You're overthinking this. The answer is very simple: if you like the paper and think it advances science in some meaningful way, recommend acceptance. If not, recommend rejection. That's all there is to it. The ranking of the conference is irrelevant.

The point is that the conference organizers have decided to put you in charge of making a recommendation based on your taste and your standards. You are now the leader and the tastemaker (to a limited extent, since your recommendation is still subject to review by the program committee), so act on that role - lead, don't follow, which means recommending based on what you think is the right decision. And if you impose higher standards than is typical for this conference, well, then the conference will actually be slightly more highly regarded next year; it is precisely through the collective leadership of reviewers and editors that conferences and journals acquire their reputation. So don't worry about conforming to other people's notions of how selective the conference should be. Just make up your own and go with that.

  • 3
    I'm not sure it's a good idea to suggest a reviewer to not base their decision on the standards of the venue, as it makes the reviewing process even more random and potentially unfair than it already tends to be. Since the OP is an exceptional situation, in which they are not aware of the standards of the conference, I also strongly object to the idea that their decision will lead to an actual improvement in the reputation of the venue - if anything, it will lead to a single group of authors being disappointed about a reviewer who applied an unreasonably high standard. – lighthouse keeper Mar 23 '17 at 21:31
  • "The point is that the conference organizers have decided to put you in charge of making a recommendation based on your taste and your standards." Did they? Or did they do a bad/lazy job of conveying the standards to be used? (Or did they do a good enough job with that but choose a referee that doesn't have the expertise to implement these standards?) "Just make up your own and go with that." I agree with @lighthouse that in certain situations this could end up causing harm, e.g. relative to backing out of the refereeing job. – Pete L. Clark Mar 23 '17 at 23:16
  • @Pete I stand by what I said. It is the job of the conference organizers to choose qualified referees and to give them good guidance about what's expected of them, and it is the job of the referees to give the best recommendation they can, incorporating the guidance they receive and their personal views of what they think counts for good work. Any harm that may be caused by poor guidance from the organizers is 100% their fault and need not concern OP. The only caveat I would add is that if OP feels truly unqualified to provide a useful recommendation then they should let the organizers know, – Dan Romik Mar 24 '17 at 0:36
  • 2
    @DanRomik I think your mode of action would indeed be fair if you were the only reviewer at the conference, since this would mean that every author group is assessed by the same standard. But when you have multiple reviewers who apply different standards, I feel it's unfair to the authors who are less lucky with the reviewers. – lighthouse keeper Mar 24 '17 at 7:46
  • 1
    @lighthousekeeper ok, we've both made our points I think. Let's agree to disagree. – Dan Romik Mar 24 '17 at 8:08
-3

As I understood, your confusion is this: There is a paper X which has been submitted to conf-A and you have to review it and give the recommendation. Fact is that rank(A) < rank(top-tier). I am a CS guy and can answer this with my somewhat philosophical way.

I have reviewed many works till now, and in my experience, I never incorporate the rank (or, popularity) of the venue (say it journal/conference) into my review quality i.e. the recommendation.

We all are doing scientific duty, we are laborers of Science. We just have to work on our assigned duty. It would be really wise to do that duty with right commitment and without compromising on the quality of the outcome.

IMHO, it would be better if you could review the paper just as a scientific work, rather than focusing on the venue ranking etc. Give your best review and let the conference committee take the final call.

The computer science research has degraded recently because of some miscreant venues such as bogus journals and conferences. So, it is high time that all researchers do their duty (work, review, edits etc.) honestly and with high quality.

  • 4
    I wish it were that easy, but I know that when people are aware of the relative competitiveness of conferences, there are many times when people say "this paper wouldn't get into XXX but it is acceptable for YYY". I don't think it's fair to say that every conference should have a universal level, but even if you desired such an outcome, I don't see how you get there overnight. That being said, individuals may recommend rejection from YYY because they are applying the standard from XXX -- not everyone has the same scale. But in practice, most grade on a curve. – Fred Douglis Mar 23 '17 at 18:01
  • I feel this answer avoids the initial question: Given a mediocre (neither particularly good or bad) paper, how to decide for the recommendation? – lighthouse keeper Mar 23 '17 at 21:17
  • 1
    I agree with @lighthouse keeper. In fact, I am not confident that the answerer fully understood the question nor that I really understand the answer. E.g.: "We all are doing scientific duty, we are laborers of Science. We just have to work on our assigned duty. It would be really wise to do that duty with right commitment." Sure; what else? "and without compromising on the quality of the outcome." Que?? The outcome is either acceptance or rejection. So how would a referee compromise the quality of it? I just don't understand what the words mean in this context. – Pete L. Clark Mar 23 '17 at 22:21
  • I understand the intent of the answer to say that the reviewer should apply an absolute standard. Is the paper above some threshold - for instance, does it have any obvious flaws, is it repeating other work, other things that would say it shouldn't be accepted regardless of venue. I think that answer had a lot in common with mine, when I said to do a detailed review providing feedback on what is good and bad about the paper. The difference is, when actually asked to commit yea or nay, I think the rating needs to be on a curve; Coder doesn't. – Fred Douglis Mar 23 '17 at 22:30
  • 1
    @Fred: That was the part of the answer that I did understand; it's the other stuff (and how this interacts with it) that I am confused by. However not taking into account the quality of the venue as a referee could only be the right thing to do if (as seems very unlikely) one is working in a field where all venues are viewed as being equally prestigious or one has only ever refereed for venues of a roughly similar quality (perhaps without explicitly realizing this). – Pete L. Clark Mar 23 '17 at 23:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.