I've been thinking about a way that researchers in a small community could stay informed about each other's work and commit to give each other feedback on preprints independently from formal conferences. (I'm thinking of a group of 10-20 researchers, who work in a common area and mostly know each other already; could be a research team in a university, or a group of people collaborating on a project.) Maybe it's the season, but it looks a bit like a secret Santa. :) My question is whether similar mechanisms already exists and have established names that I could use to think about how to organize it.
I'm thinking of something like this, with a calendar fixed in advance:
- Signup phase (day D): everyone interested signs up and submits a paper for the group. It would ideally be a recent preprint, or otherwise recently accepted work on which the submitter would like feedback. Typically you'd want the preprint to be publicly posted (e.g., on a preprint server like arXiv) to ensure there are no worries about stealing each other's ideas.
- Bidding phase (day D+7): everyone who submitted sees the list of submitted papers and bids for the papers they would most like to review.
- Assignment phase (day D+14): everyone gets one paper assigned. The assignment of who reviews whom can be public or not depending on what people prefer.
- Review submission phase (day D+28): people submit a review of their work, like they would have done if asked to peer review it. Since there is no question of accepting/rejecting papers, reviewers can focus on comments, suggestions for improvement, points they found unclear in the paper, etc.
- Sharing reviews: once a participant submits their review, they receive the review that was submitted about their own work.
My question: do similar arrangements already exist? have people successfully set up such mechanisms, or written about their experience with such mechanisms?
Here is how it differs from similar systems in academia:
- This is essentially like a small-scale conference or workshop, but without an in-person meeting, and without any question of formal publication of the papers, so this can be independent from standard peer reviewing. Also, these reviews can be done without worrying about conflicts of interest (many of the participants would be in COI anyways), so participants can bid on the submissions they really want to read, even if they are written by people they are in COI with.
- Of course many people already send preprints of their works to colleagues and other collaborators, but usually with no strong expectation of receiving detailed feedback. I don't know of people with explicit reciprocity agreements "you read my work, I'll read yours".
- I know about reading groups, book discussion clubs, and journal clubs. But these are typically about discussing specific works in depth as a group. Also, typically these are about reading works from people outside of the group, not about giving/receiving feedback within the group.
- The closest I can find is preprint discussion or reviewing systems like Peer Community In. But these aim to have a large scope like a disciplinary journal, not to be an informal small-scale affair.
- There seem to be this kind of reviewing circle arrangements outside of academia, e.g., Critique Circle for writers.
To give some context: my field is theoretical computer science, but this proposal seems pretty generic and would seem to be applicable to any other field where people typically post their work as a preprint. My motivation for such a system is that I find it hard to read the papers of people from my area (even when it's close to my interests), and hard to get feedback about my own work. The exception is standard peer review, where I manage to commit to a deadline and read in depth a specific paper: but there I typically end up with papers I don't care so much about, and I miss all the relevant work from people that I'm in COI with.
