3

I was reading this article on The Guardian about scientific publishing and peer-review process. The author raises a fair number of concerns with the existing model, most of which I agree with. The article ends with the following statement:

... but the need for reform is profound.

I ended up thinking about this idea, a reform in the way we do science. By doing science I mean the whole process of writing, submitting, reviewing, publishing and measuring the results of our work (and not the actual work itself).

While I agree with the author about the current system being faulty, sick, or corrupt (depending on the shade of glasses you look at it), I cannot really come up with a system that would magically solve the existing problems just like that.

The Arxiv idea is named in the article and from what I understand people in physics and math quite enjoy the benefits of that site (whatever it is) however with the amount of money and prestige involved, I cannot imagine how it would be implemented in fields like biology, chemistry and medicine.

To sum things up, my question is whether or not there are/have been alternative models to the existing one: blind, peer-review, hundred of journals, and metrics of "quality" flying left and right. If so which shortcomings of the existing model were amended?

  • 11
    Arxiv is not peer-review. – GEdgar Aug 23 '15 at 18:33
  • @GEdgar that is not a helpful comment. If my misunderstanding of a platform that I've never used is important enough for you to comment on, you could have at least corrected my mistake. – posdef Aug 23 '15 at 22:08
  • 5
    That is a helpful comment. Its purpose is precisely correcting your mistake. What should he have done instead? You would have reacted worse to an uncalled-for edit to remove that statement. And please don't get offended because of the correction: when people correct you, that doesn't mean they think you're an idiot. Everyone is wrong sometimes. :) – Federico Poloni Aug 24 '15 at 7:27
  • @FedericoPoloni It's not that I am offended, and I have no illusions thinking that i'd be right all the time :) I just think that it's a besserwisser comment, it points out an mistake and makes no effort in improving it. I would have been thankful if the comments was something like "Arxiv is not peer-review, but rather ..." – posdef Aug 24 '15 at 8:06
  • I think this is unclear, far too broad, and answers will probably be opinion-based. You haven't even stated what the "problems" are, how they relate to publishing models, and each separate problem may or may not be solved by differerent new publishing models. – fileunderwater Aug 24 '15 at 10:06
6

As I see it, the issue is not actually how publishing and reviewing work. There are lots of alternative models out there, including modifications of blinding (e.g., Frontiers names reviewers when a paper is published), alternative approaches to peer review (e.g., IETF RFCs, math and physics use of arXiv), and decreased need for additional journals by removal of artificial "throttling" of publication rate (e.g., PLOS ONE).

Instead, my belief is that the problems we face lie more in the way in which quantification of publications is used as a proxy for evaluating significance and impact of work. Peer review is a reasonable system, and many of the tweaks on it look like fine systems as well. Making peer review high stakes, however, turns it into a system that people need to try to game and distort because it directly impacts their ability to remain employed and to fund their research.

Because of this, I think that any attempt to "fix publishing" will not actually fix the problem: at best, it will simply shift the focus of high-stakes distortion to a different area. This is something that we see already, with the emergence of this great diversity of different "quality metrics," each of which is gameable in various different ways.

At the end of the day, however, all decisions about scientific funding boil down to some mixture of personal judgement and some sort of metrics. The more that we have metrics (publication or otherwise), the more that the system can be gamed; the more that we rely on personal judgement, the more subject the system is to bias, prejudice, and cronyism. I don't have any clear notions on how best to go about improving the balance in order to improve things, but I think it's important to have a clear identification of the actual locus of the problem that needs to be addressed.

  • Are you saying that the problem is not with the publishing system but with the academic hiring/promotion system? – BrenBarn Aug 24 '15 at 2:09
  • 2
    @BrenBarn ... and funding too. – jakebeal Aug 24 '15 at 2:20
  • 1
    In that case, +1! :-) – BrenBarn Aug 24 '15 at 2:25
  • (Genuine question) What, in your opinion, would make for good 'quality metric'? Some abstract notion of how much 'impact' the person's work has? – DTR Aug 25 '15 at 12:33
  • 1
    @DTR I think that the notion of a single quality metric is fundamentally mistaken, since there is not a single type of scientist or scientific success. I think that we would do much better with a spectrum of quality metrics that can help identify which paths to scientific impact a person has taken, and how much impact they have had in those ways. These are not commensurate, and blending into "one big metric" is just as incorrect when evaluating a mid-career scientist as it is when evaluating an application for university or grad school. – jakebeal Aug 25 '15 at 13:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.