15

This question already has an answer here:

What are the ethics of publishing (on the web, etc.) peer reviews received for a paper? Does your answer change if the paper was accepted or rejected?

I am specifically interested in the "one-shot" case typical of computing science conferences, without rebuttals, where there is no or limited dialogue between an author and the reviewers.

This is quite distinct from Can I publish the reviews I write? as here I am talking about reviews I have received, not those I have written - the reviews' authors retain their anonymity, and presumably the paper would be included alongside those reviews.

Vijay's response below includes a summary of much of the other responses and my comments on those.

marked as duplicate by Austin Henley, F'x, scaaahu, Peter Jansson, Nicholas Jul 23 '13 at 8:58

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

  • 2
    What are you trying to achieve? Why would you publish the reviews, what would you gain? If you have a certain objective in mind, perhaps it would be better to ask a direct question about how to achieve this objective? – Jukka Suomela Jul 15 '13 at 18:19
  • 1
    I empathise with the OP. Having received reviews that were incorrect, lazy, territorial, or plain vindictive, I feel frustrated and helplessly trapped in a system which has no incentives to rectify itself and has a dissatisfyingly asymmetric dialogue structure. Publishing reviews are the only outlet I can think of. I can also leave academia, which will mercifully make the problem go away (for me). – Vijay D Jul 16 '13 at 4:53
  • 2
    @Peter: This is not a discussion forum, this is a Q&A site. You should have a clear, practical question that can be answered, not an open-ended invitation for opinions and discussions. – Jukka Suomela Jul 16 '13 at 12:27
  • 1
    @Peter, stackexchange convention is to improve your original question following feedback. You could add clarifications about your feeling the review process has flaws or poor quality, that you are from a computer science background, that the reviews you intend to publish are anonymous, etc. Also these sites are for questions that can be answered from existing knowledge, not which require new research and investigation. – Vijay D Jul 16 '13 at 21:01
8

Let me add some data points to this discussion. Some items below are about discussing reviews and editors in a public forum and not about publishing reviews.

  • Jens Teubner makes available the reviews to his papers online. I do not know him but I have heard from someone who does that he said he has not received negative feedback about doing this. Maybe you can ask him for more information.
  • Doron Zeilberger has published reviews in Opinion 87 and for his paper Automatic CounTiling.
  • Doron Zeilberger's Opinion 61 is about rejections and accountability (it has a response from Luca Trevisan) and his Opinion 81 is about rejection and snobbery.
  • Peter Clark's mail to Doron Zeilberger about some material Zeilberger published online. This is only tangential to your question, but I think it's good to keep in mind that when you go down the road of open publication, you should be ready for others publishing material about you or your reviews.
  • The Writings of Leslie Lamport chronicles in very direct terms the stories behind his papers including some notes about editors and reviewers (for example papers 62, 122,129, ).
  • In The Writings of Leslie Lamport Paper 132 he talks about having written an 'unkind review'.

I have been meaning to publish my reviews for multiple reasons and I am glad to see that other people have been thinking the same and some have done it. I do not think the result or answer should depend on whether the paper is accepted or rejected. If reviews are published, I believe that one should also publish the version of the article that was used to make the reviews. Otherwise, the reviews are like quotations taken out of context. If there are coauthors, one should obtain their permission first, or at least include a disclaimer that you are publishing your reviews on your behalf only. As an example of a disclaimer, see Lamport's page:

Some of the stories read like complaints of unfair treatment by editors or referees. Such cases are bound to arise in any activity based on human judgment. On the whole, I have had little trouble getting my papers published. In fact, I have profited from the natural tendency of editors and referees to be less critical of the work of established scientists. But I think it's worth mentioning the cases where the system didn't work as it should.

The notification of acceptance is usually signed by the editor of the venue, even in the case of anonymous review, so one should ideally obtain their permission if you will reveal who the editor is. The same applies for reviewer permission in signed reviews. I cannot tell whether people in the links I give above obtained editor permission first so there may be precedent for not doing so. Publishing reviews intended for private circulation still takes them out of context. I would add a disclaimer that the reviews and notification letter were written as private communication and if published without permission of named entities, I would note that too. The latter is in case your intention is to protest the status quo by subverting standard conventions.

I haven't answered your question because I don't have a clear answer. Publishing reviews is not conventional academic behaviour. Doing so can be construed as unprofessional depending on how you publish them and what additional commentary you add. Calling it "unethical" seems a rather heavy handed judgement to make. There are things an anonymous reviewer can do that are clearly unethical (steal research, suppress publication, circulate the manuscript) or questionable depending on context (force citations, comparisons, reject without reading, write ad hominem reviews etc.). There is very little an author can do to wrong a reviewer that is remotely comparable and publishing a review does not seem remotely on that scale to me.

There are multiple reasons to publish reviews including accountability for all parties involved (including authors) and as a form of protest. I believe this intent is important to consider because forms of protest do subvert what may be considered acceptable behaviour. A useful thought exercise might be to put yourself in the reviewer seat and ask if you would be fine with similar treatment. I would not have an issue if the reviews I have written were made public (even if I knew that stylometric techniques could be used to identify me). I do not think all reviewers feel that way.

Finally, let me point again to Jens Teubner's page, which comes across to me as a model of how to publish reviews while retaining professionalism and dignity.

  • +1 for the excellent and interesting links to the blog posts! – Chris Gregg Jul 16 '13 at 7:05
  • If it is necessary to ask coauthors of the paper for permission, why do you think it is not necessary to ask the author of the review? – silvado Jul 16 '13 at 7:16
  • Thanks Vijay. :-) I would be completely happy to have my name publicly associated with the reviews I have written, and for those to be published in any form. This discussion is making me think that such publication is not unethical, merely career-suicidal. @silvado - the reviewer is anonymous and remains so. How do I ask their permission? – Peter Jul 16 '13 at 11:21
  • 1
    @Peter I don't think it is career-suicide if you do it professionally. I haven't done so yet because I cannot stop myself from ranting. If a large group of us did it, it will not be unconventional. Another option is to start a Wikileaks for paper reviews and take joy in each other's pain. – Vijay D Jul 16 '13 at 20:44
  • @Peter You can only ask for permission via the editor, but I think it is unlikely that an editor will forward such a request. – silvado Jul 18 '13 at 6:07
8

In the journals where I have been invloved as editor, author or reviewer (except in ones where the review process is open on the web), the implicit understanding has been that the communication is closed between the author-editor-reviewer. I have not seen any explicit rules stated by these journals to control against any such public dissemination but I have not looked for it either. I think that, particularly in cases where it is not explicitly stated that reviews are public, posting reviews without consent from the reviewer would be similar to publisihing someone private letters wihout consent. It is not illegal but ethically very distasteful. I do not think it matters whether or not the review was anonymous. Whether the paper was accepted or not would not matter either. I would consider posting without consent just as bad under all circumstances.

  • 3
    This is my understanding too, but I fail to see the parallel with private letters when one party is anonymous. How about if the review process does not have a rebuttal or response phase? Is it unreasonable to embarrass a (CS) conference by revealing the poor quality of its reviews? How else can we signal this? – Peter Jul 12 '13 at 13:13
  • 1
    Good points. Usualy poor practises become known without such public exposure. I would start by discussing the matter with peers who may have had similar experiences (same conf). How to later convey it to the conf. is difficult to assess at this stage. – Peter Jansson Jul 12 '13 at 13:18
  • Peter, it may be that poor practices become known within the community of the conference, but it does not make them common knowledge to the field as a whole. See my other remarks for other utilitarian benefits of author-publicised reviews (at least in the case of CS confs without rebuttals etc). – Peter Jul 14 '13 at 12:57
6

As argued in the answer by Peter Jansson, it is ethically not acceptable to publish peer reviews that you receive for your paper, and I agree with that.

But there is also a legal aspect to it. The review is an intellectual work and as such will typically be subject to copyright. And that holds even if you don't know the author. So publishing a review will be a copyright violation, unless you get permission from the person who wrote the review.

If you want to highlight bad review practices, instead of just publishing the review consider citing from it. The actual difference may be marginal, but from both a legal and an ethical perspective, you should fare much better with this approach.

And my answer doesn't change depending on whether the paper is accepted or not.

  • I'm still not seeing an argument for why it is unethical. For instance, if the reviewers make false or abusive remarks, and there is no avenue for a rejoinder, then anonymous peer review is already not very ethical. One reason to reveal the entirety of the reviews is to ensure the reader knows I am not cherry picking from them for effect. – Peter Jul 12 '13 at 23:13
  • 2
    @Peter - How can there be no avenue unless you know for certain that the people writing the reviews are organizing the conference/in charge of the journal? There's usually a hierarchy of people to complain to. – Rex Kerr Jul 14 '13 at 17:43
  • @Peter I think I understand your intention. In that case, there may also be ethical arguments for publishing (as there are for keeping the reviews confidential). If it is some sort of "whistle-blowing", that's often a difficult case-by-case decision. If you are specifically interested in that, why not ask a specific question? – silvado Jul 15 '13 at 6:49
2

If the question is "Should the author publish the reviews he got for his paper(s)?", then I'd say no (maybe with some exceptions, but this would have require a per-case discussion). I believe the other answers have more or less covered that.

But, if the question is "Should the journals publish the reviews?", then I believe it would be both ethical and useful to publish the final positive reviews, and I have several reasons for this.

First, giving a positive review is like giving a positive grade on an exam. The person doing so should stay behind his "verdict" with his professional reputation. The review is one of the results of the work that the researchers do, so publishing positive reviews doesn't seem to me much different from publishing the results of the research in papers.

Second, and quite related to the first, I've read some really crappy papers, with nonsense, obvious errors, misquotes, etc. Reviewer cannot "catch" everything, but some of the papers get bad enough that it is obvious that the reviewer didn't do his job. If the reviewers knew their names would forever be publicly associated with such paper, I believe some of these might actually try to do their job.

As for the negative reviews, I see no point in "shaming" the author if his paper was too bad (in whatever way) to be published. If this was not the case, but the reviewer is to be "blamed" (i.e., for misunderstanding the paper), the issue can be resolved with the editor, or the paper can be submitted elsewhere, again giving no reason to make the negative review public.

One might argue that the negative review is also like publishing the results of the research in papers, but I see it more like a failed research, which is not something that one usually publishes.

  • Please read my comments to the other responses. I know the protocol; I am not expecting reviewers to shed their anonymity; I am not talking about journals that have a robust rebuttal/response process. If you want a concrete proposal, consider this: the author has the discretion to publish the (anonymous) reviews of their paper, whether it gets accepted or not. What is ethically wrong with this? What would be wrong with publishing failed research anyway? It'd save the rest of us some time, in many cases. – Peter Jul 14 '13 at 5:50
  • O.K., I have reread them now, and the questions here made some clarification as well. I am probably not the best person to answer the ethical aspects, since I always vote for the transparency and I would be O.K. with this (even with my name included in the review, although I am aware that you are not asking about that). However, I see two practical problems: 1) no one, or at least almost no one, would be interested in reading/analyzing reviews for other people's papers; 2) the purpose, beyond "badmouthing" a paper/conference, eludes me. – Vedran Šego Jul 14 '13 at 11:43
  • If someone chooses bad reviewers, this will be reflected on the quality of the publication, which I'd expect to affect that paper's/conference's reputation far more than publishing some reviews that almost no one would be interested in reading. Also, it would probably be seen as a personal attack on the paper/conference, which may cause you trouble when trying to publish in the future. The research community is not above the petty "common human" flaws. To summarize these two comments: I'd advise against it, but for the practical rather than the ethical reasons. – Vedran Šego Jul 14 '13 at 11:43
  • Vedran - thanks for your comments, but as I've remarked elsewhere, my interest is in an argument for why revealing reviews is not ethical. (The venue need not publish the reviews - I am suggesting that the authors of the paper have the discretion to.) As for utility: it gives some indication of what it takes to get accepted, how friendly and constructive the community is, and so forth. It also encourages civility and defensible criticisms. Etc. etc. – Peter Jul 14 '13 at 12:00
  • As I said, I agree that it should be perfectly O.K. and I don't see it as unethical, but I also have a feeling you'd get quite a resistance (probably not even the open kind, but the one behind your back). I'd also expect such revealing to be labeled "unethical" by many, but without the answer to your "why is it unethical?". After all, that's what the labels are for: avoiding real arguments. – Vedran Šego Jul 14 '13 at 12:43
1

The ethics are the same as publishing any other communication that is assumed to be in confidence: if the content contains critical information about an illegal activity, you may (depending on the nature/severity) be morally obligated to turn it over as evidence to an appropriate party; if it is otherwise urgent or critical for others to be aware, you may be morally encouraged to find a venue to publicize it (as "whistleblower"), and in pretty much all other cases, you should keep in private.

You can complain to editors and others responsible for the conference or journal; you can commiserate with colleagues and try to find out whether this is systematic or not. But trying to shame reviewers in public is very unlikely to accomplish any valuable goal. (It may provide a satisfying revenge at the cost of other of your goals; I encourage you to think very carefully before deciding that this is worth it.)

There are all sorts of reasons you might have a bad review (covered in other answers); assuming one that makes you the most angry and/or feel most self-justified is a very natural reaction, but probably isn't the best way to a productive conclusion.

Instead, when you find things like this, you are probably much better off applying to other conferences, publishing in other journals, and maybe quoting (anonymously) the worst parts of the review in a blog or somesuch explaining why you're no longer going to whichever conference or considering publishing in whatever journal. There are options in some fields for publishing with open reviews (the Frontiers life science journals, for instance).

Publishing the full review and naming names is a good way to make people angry, make other reviewers not want to review your work, and to not get anything to change. Adding your voice (and papers!) to existing movements that seek to improve the peer-review process is much more productive, even if it's not as viscerally satisfying.

To be completely clear about the ethical implications: you are breaching the trust of the editor(s)/organizer(s)/reviewer(s) that reviews are confidential information. Breaching trust makes people less likely to trust you in the future, and is (in most ethical frameworks, including intuitive ones) unethical unless perhaps there is a very compelling case why this must be done. In your situation you have not articulated such a compelling case, especially given that are a variety of other avenues to take if you're actually concerned with the quality of the reviews. If you are working in academia, presumably you know how to quote tiny sections of a full work to make your point--that's much more acceptable.

  • I don't think you have addressed my question. The reviews are anonymous; I cannot name names. I am not looking for advice; I know the protocol is to hold reviews in confidence come what may. I am asking for what the ethical implications would be of violating this protocol. See my other comments for further details. I should add that I am specifically bothered by the low quality of reviews (which I cannot demonstrate to you without violating the protocol!) and not the accept/reject recommendation. It is not helpful to make this an emotional issue. – Peter Jul 14 '13 at 5:45
  • @Peter - I added another paragraph making explicit the ethical implications that I was implicitly referring to in the rest of the answer. – Rex Kerr Jul 14 '13 at 17:41
  • I'm not convinced by this normative argument - it is just a statement of convention. I expect to be held accountable for the contents of my papers; why should reviewers not be held to the same standard? As I mentioned before, selective quoting of reviews may not be very convincing - of course I'll choose the egregious parts and then be charged with cherry-picking and ignored. Would your answer change if the reviewers knew up-front that their words may be made public by the author? (See my other comments) – Peter Jul 14 '13 at 22:50
  • @Peter - Ethics in such matters is based on convention (as is what constitutes abusive language, for instance). If reviews are not confidential information, then you are not breaking convention. If the reviewers knew you were probably going to break convention and publish the reviews anyway, then no, my answer would not change, just like I wouldn't say that it was ethical of you to steal pies from someone's window where they were cooling just because they knew you might. To hold reviewers accountable you talk to the editor or organizer. – Rex Kerr Jul 15 '13 at 3:05
  • Sure, ethics have some foundation in convention. I'm wondering if the convention can or should be changed. If the reviewers knew that authors had the discretion to publish reviews (i.e., adopting this as a new convention), what other ethical considerations would stop you from publishing them? – Peter Jul 16 '13 at 11:30
0

As said before, publishing reviews breaks convention but this, in itself, does not make it unethical. Publishing anonymous reviews does not really affect reviewers—reviewers could do much worse to authors than what authors could do to reviewers. (See Vijay's answer for examples.)

In my opinion, the confidentiality of reviews is to protect the author: the fact that a paper got submitted remains confidential prior to publication. Thus, authors may considerably rework their paper after receiving peer reviews, resubmit to another conference or even completely abandon it.

Reviewers are protected by their anonymity -- such that a young researcher can openly state his opinion about the paper of a renowned academic without fearing bad consequences. I'd assume even stylometric analysis cannot reveal a reviewer's identity without doubt.

Despite all that, publishing reviews might still be regarded as unprofessional, depending on circumstances and context.

-2

This might be related to your question: http://www.peerageofscience.org/

The concept is that 1) you peer-review your paper before submitting it to a journal, 2) peer-reviews are peer-reviewed, 3) participating journals send you publication offers if they like your paper, 4) you may submit your peer-reviewed paper to any journal you like

I'm not sure are the reviews public themselves, I did not use the site yet, but at least there are peer-review scores appearing on the main page

  • If you're a reader of Science, also check this out: sciencemag.org/content/335/6067/385.1.summary – just thinking Jul 12 '13 at 18:38
  • Why the down-vote? – just thinking Jul 12 '13 at 18:58
  • 2
    I just don't see how it answers the question. – silvado Jul 12 '13 at 19:15
  • Well, maybe you are correct. It does not really answer the question directly. The reason I posted the URL is that the service is taking a step towards what the OP is asking (open peer-review if I see the question correctly) – just thinking Jul 12 '13 at 19:26
  • That would be better suited as a comment then. Or try to relate it to the actual question then. – silvado Jul 12 '13 at 19:31

Not the answer you're looking for? Browse other questions tagged or ask your own question.