Towards a Stackexchange-like comment/reputation system for research papers

Question

We have had quite a few discussions on how to gauge the quality of research papers and academic journals. The consensus is that it is tough to identify a single metric which could reveal this quality factor.

I had an idea in this context. In the past newspapers used to have a single "Letters to Editor" section to which comments on all articles were sent. With the advent of the Net, we have a comments section under every article.

• Similarly, why don't reputable journals introduce a comment space for their published papers, where fellow researchers could appreciate/criticise/query the works?
• What hinders us from having a reputation system for published papers - something where registered users could vote based on their perceived utility?

Answer 1

Similarly, why don't reputable journals introduce a comment space for their published papers, where fellow researchers could appreciate/criticise/query the works?

They don't need to create such a space, since these comments can already appear elsewhere on the internet. Nevertheless, some journals have tried, but they typically attract very few comments, and nobody knows how to create a viable community this way. See that link (and the blog posts it links to) for further discussion.

What hinders us from having a reputation system for published papers - something where registered users could vote based on their perceived utility?

One major obstacle is that not enough people seem to want to use such a system. There have been many attempts to set one up, but none have caught on.

However, there's a deeper obstacle. In order to get any meaningful results at all, you need to solve at least three problems: ensuring that votes come only from competent researchers, ensuring that nobody can deliberately manipulate the system or cheat, and ensuring that voters have the right incentives even if they are not trying to be dishonest. In principle, it might be possible to solve these problems, but it would take a nontrivial infrastructure. However, the more elaborate the system becomes, the harder it will be to get lots of users. A large majority of researchers won't pay any attention to a system like this, and some of the ones who do care will be strongly opposed, so getting widespread adoption will be an uphill battle.

Answer 2

As a follow up to Anonymous Mathematician's answer, I think the answer lies in Arxiv: a lot of people already use it, there is no profiting involved and even if a paper is moved to a given journal, the paper's page main remain online with all discussions In it (except that the PDF is replaced with a link). So it can create a solid up/down voting system.

As for reputation, everyone may start equally but a Web of Science citation analysis,for example, may be used to yield a larger starting reputation to get things going.

Going further, this may also replace the reviewing process: the paper is uploaded, everyone may read it and someone who thinks it is interesting may then apply as a candidate reviewer. This would avoid reviewers who don't really understand about your work (as is certainly not uncommon to happen).

It is a leap of faith to trust a reputation system rather than a faculty hierarchy. Notwithstanding, note that the current review system is already embalmed in such trust: even a high school student can publish a paper if it is reviewed to be interesting; conversely, if you publish nice papers you are invited to review nice papers.

In a sense, this shifts the focus away from the periodicals themselves.

Answer 3

For an update, you might want to have a look at probably the largest organized group discussing this and similar issues - FORCE http://www.force11.org/ - stands for "The future of research communication and e-scholarship"

In Manifesto they say that new models of reputation are required, etc. However, a ready to use system is not yet developed

Answer 4

When I submit a manuscript, I value the work of the reviewers a lot. Revision work is, most of the time, an excellent exercise that allows you to question your own work, and improve it. In a way, getting my article accepted in a reputable journal is one of the few signs that what I do is of some quality (or not, if it is rejected). So, peer review is a system in which people 'vote' for your work, and suggest edits, similarly to SO Q&As. When people cite my work, it is the equivalent of the upvotes on SO. Of course, exactly like on SO, the more popular your domain is (your academic 'tag'), the easier it is to get upvotes. But people you care about are usually clever enough to figure that out.

Now, is peer-review a perfect system? No. But neither is SO's voting system. At the end, what one needs to do is use one's brain when reading something: you are probably not too bad at judging quality of the work published in your field.

This being said, I think that it is a good idea to have an alternative system of evaluating research. I find that the discussion of articles on pubpeer.com to be a good way to anonymously criticize published literature.

I should add that I have the chance (or was it a choice..?) to work in a field where there is little political involvement. So I could imagine that comments of readers would be for the most part constructive and in the worst case, irrelevant. But that is not the case for everyone...

Answer 5

Also complementing Anonymous Mathematician's answer: one of the main uses of the "peer-review system" is "allocation of a scarce resource", namely, academic positions and grants. For that matter, it is not really accurate to talk about "peer review" in that regard, because typically it is not one's peers but one's "seniors" or "competitors" that reject one's paper, whether or not with good reason. But it is pleasanter to talk about "peer review" than "gatekeeping".

Another corruptive influence is that journals themselves vie for "reputation", which drives an odd jousting-for-position between authors and editors, wherein each tries to acquire as much status as possible, by connecting to higher-status "others"... Let's ask ourselves whether the same impulse wouldn't corrupt nearly any publication-for-status structure. I have no idea how to design-against, or design-around this.

If the status rewards are reduced beyond a certain point, we might indeed find ourselves with the "hermit gatekeepers" who simply have nothing better to do... as mentioned in the comment by John Moellner. Given human nature, it seems that the broad pattern demands incentives that are not entirely idealistic.

As it is, we can easily see that on Math Overflow, MathStackExchange, and many such vehicles, the most-popular questions are approximate repeats of questions that have been asked many times before. For a year or so, such a site will have a feel of freshness and novelty, but it is not clear to me what happens after some years when the limited repertoire of basic questions (motivated by undergrad or grad courses and such) is fairly completely documented, so that one no longer asks a question, but first carefully does the archival search.

I would advocate that academics who are already established, tenured, etc., think more in terms of the subject matter itself and putting helpful things on-line, rather than continuing to take up quite so much space in the game that beginners must play to get a position, get tenure, etc. True, one won't maximize one's pay raise each year without continuing to play the same game forever, but academic departments could really work harder to figure out how to appraise post-tenure people, which wouldn't upset the tenure criteria per-se at all. "Get out of the young peoples' way."

Edit: In response to "comment": I think the sense in which these points address the original question is that, if there were no presumption that the present "peer review" system should be supplanted, then why ask the question at all? Just use the "stack exchange" or similar software and everyone who wants to put papers or reviews of papers on it, and vote, or down-vote, can do so. There is no obstacle to this, so why ask the question... except to propose also that there be a transition. My remarks above, and some of the other comments, indicate why I think there would be problems. E.g., many of the people one would want to participate would not, for a variety of reasons, which would certainly have the effect of failing to confer one sort of status of a sort currently essential to professional survival at a junior level.

That is, putting the question in a context in which it makes sense to ask it (rather than a more formal sense somewhat disjoint from that context), the other parts of the context prove to be significant in answering the original. If one speculates that the issue is about replacing "peer reviewed traditional journal publication" (paper or electronic) with a different structure, then it is reasonable to see why there would be trouble.

Answer 6

First, I rely on the response of others here regarding nuanced comments. I take it they're saying that if we do facilitate appreciation, critique, and query within a journal, and even if we don't, those kinds of responses will flourish primarily outside the journal.

Second, here is a notable hindrance to a reputation score system for published papers. How you respond to this hindrance constrains how you design, or refuse to design, such a system.

A voting system is a systematic decision about ranking multiple options that is based on multiple individuals' recommendations for ranking those options.

Trick question: Do we accept that definition? Because, if so, the problem is that no satisfactory voting system can exist.

Arrow's impossibility theorem states that every systematic ranking based on the rankings from each of a group of recommenders is unsatisfactory in at least one of these ways:

• The group's ranking is completely controlled by one single recommender. Comment: we could call this person the "dictator", the "editor-in-chief", or as @JohnMoeller said above the "hermit gatekeeper".
• The group's ranking sometimes orders two options differently from the ordering that was chosen by every single one of the recommenders.
• The group's ranking sometimes orders two options differently, say, A and B, because recommenders changed their recommended rankings overall, even though every single one of the recommenders kept unchanged their own ordering of A and B.