4

I've been thinking about a way that researchers in a small community could stay informed about each other's work and commit to give each other feedback on preprints independently from formal conferences. (I'm thinking of a group of 10-20 researchers, who work in a common area and mostly know each other already; could be a research team in a university, or a group of people collaborating on a project.) Maybe it's the season, but it looks a bit like a secret Santa. :) My question is whether similar mechanisms already exists and have established names that I could use to think about how to organize it.

I'm thinking of something like this, with a calendar fixed in advance:

  • Signup phase (day D): everyone interested signs up and submits a paper for the group. It would ideally be a recent preprint, or otherwise recently accepted work on which the submitter would like feedback. Typically you'd want the preprint to be publicly posted (e.g., on a preprint server like arXiv) to ensure there are no worries about stealing each other's ideas.
  • Bidding phase (day D+7): everyone who submitted sees the list of submitted papers and bids for the papers they would most like to review.
  • Assignment phase (day D+14): everyone gets one paper assigned. The assignment of who reviews whom can be public or not depending on what people prefer.
  • Review submission phase (day D+28): people submit a review of their work, like they would have done if asked to peer review it. Since there is no question of accepting/rejecting papers, reviewers can focus on comments, suggestions for improvement, points they found unclear in the paper, etc.
  • Sharing reviews: once a participant submits their review, they receive the review that was submitted about their own work.

My question: do similar arrangements already exist? have people successfully set up such mechanisms, or written about their experience with such mechanisms?

Here is how it differs from similar systems in academia:

  • This is essentially like a small-scale conference or workshop, but without an in-person meeting, and without any question of formal publication of the papers, so this can be independent from standard peer reviewing. Also, these reviews can be done without worrying about conflicts of interest (many of the participants would be in COI anyways), so participants can bid on the submissions they really want to read, even if they are written by people they are in COI with.
  • Of course many people already send preprints of their works to colleagues and other collaborators, but usually with no strong expectation of receiving detailed feedback. I don't know of people with explicit reciprocity agreements "you read my work, I'll read yours".
  • I know about reading groups, book discussion clubs, and journal clubs. But these are typically about discussing specific works in depth as a group. Also, typically these are about reading works from people outside of the group, not about giving/receiving feedback within the group.
  • The closest I can find is preprint discussion or reviewing systems like Peer Community In. But these aim to have a large scope like a disciplinary journal, not to be an informal small-scale affair.
  • There seem to be this kind of reviewing circle arrangements outside of academia, e.g., Critique Circle for writers.

To give some context: my field is theoretical computer science, but this proposal seems pretty generic and would seem to be applicable to any other field where people typically post their work as a preprint. My motivation for such a system is that I find it hard to read the papers of people from my area (even when it's close to my interests), and hard to get feedback about my own work. The exception is standard peer review, where I manage to commit to a deadline and read in depth a specific paper: but there I typically end up with papers I don't care so much about, and I miss all the relevant work from people that I'm in COI with.

3
  • 4
    It seems like a lot of effort for little benefit. And it would be difficult to guarantee anonymity in any case since people would already know one another. Why not just emphasize collaboration?
    – Buffy
    Nov 26, 2023 at 19:32
  • And, if people with a COI can bid on and review an article, how can you trust any review?
    – Buffy
    Nov 26, 2023 at 20:06
  • People used to do this as a one time thing via Twitter for job market candidates… but not sure if that has continued …
    – Dawn
    Nov 27, 2023 at 14:01

2 Answers 2

4

PREreview is the closest thing I can think of, and most specifically their PREreview Clubs. It might be worth reaching out to them.

1
  • Thanks, I didn't know about this and it seems indeed very relevant. It looks like one difference is that they seem to focus on the club formally publishing reviews, as opposed to sending them privately to authors. Still a very interesting pointer.
    – a3nm
    Nov 27, 2023 at 18:07
-1

Expanding on my comments in a Frame Challenge. And no, I don't have experience with it, nor do I expect there to be many with such experience.

This seems to be an attempt to reproduce a journal's single blind review.

However:

Someone will probably need to know what the author-reviewer linkage is and it is possible, even likely, that pressure could be applied to them to reveal things that would not be revealed by an editor who has something to offer beyond the review itself: publication.

If someone with a COI could bid on a paper without revealing it, or be assigned a review in spite of it, then no review could be trusted. Both overly optimistic and overly pessimistic reviews could easily arise.

If the community is small then people know one another and and probably surmise who the reviewer is. This could lead to conflicts if one depends on anonymity that isn't in place.

I suspect that, people being busy, this will be a low priority for some, both in bidding and in completing reviews. It would then be difficult to maintain.


In place of this, and already pretty standard, is that people simply request feedback on their work from a trusted colleague. If the person has conflicts that is already likely known and can be accounted for. If a person is too busy, they can just say so. They can offer to swap papers for review. All in the open.

Within a large university there are often working groups within subfields. These include both professors and students. Sharing papers and work in progress is pretty common. You get the feedback you need from people you already trust. The suggested framework in the question seems like a poor substitute for this.

Moreover, in the wider world it is also pretty common for academics to have working relationships with others outside the university. Ideas and papers can be shared and feedback can be given. In larger, better funded, places, visiting for a few days or weeks... can provide an opportunity for sharing ideas and getting feedback. This already happens. Early career academics are encouraged to form such associations, perhaps by joining the circle of their advisor(s).

2
  • 1
    Thanks for the answer. About "this already exists": I know of working groups and informal sharing of work in progress. But I have never seen people in this context (including me) who would commit to reading each other's work with the same reliability as most people (including me) do for peer review towards strangers. About COIs etc., the point is to give/receive feedback from collaborators, I don't see grave implications if a review turns out to be bad (or maybe insincerely positive). I think of it like "sharing papers and work in progress" but with more structure and with commitments.
    – a3nm
    Nov 26, 2023 at 22:58
  • 1
    About "people are busy", which I agree is a big challenge on anything new in academia: my observation is that we spend copious time giving feedback to strangers via peer review, even though in my experience this is often less interesting to me as a reader, and leads to less interesting feedback, than if I were to spend the same attention on papers from colleagues and collaborators. So why are we currently so generous with our time for formal peer review, while we seem to think it is less important to read in details what the people close to us are doing?
    – a3nm
    Nov 26, 2023 at 23:01

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .