20

I was watching this video on YouTube (this is the second part of it) and it motivated me to ask this question.

From the videos:

....that's the good aspect of peer review. It should work to sift out problems with the interpretation, problems with the results. It should work to improve a paper. The problem is ..... there is a huge volume of stuff and we are increasingly getting swamped. .... here is one of the worst examples of where peer review has entirely failed. ....

The examples he talks about is discussed in this blog post.

enter image description here

When you zoom in, you can clearly see that it is photo-shopped.

...this is one of the top ranking journals in the fields...

Now here is the really troublesome aspect... if they had done it better (you can see a much better photo-shopped image in the blog), how would we know. Not only is money spent in that lab, other groups would chase this up and that's the worrying aspect and it builds and builds. Science is like that.

In the past if the referees didn't pick it up, that was the end of the story. We now have something called the post publication peer review. There are sites ( PubPeer) and there are blogs (ChemBark, Chemistry Blog) where people upload paper and say well this looks like a great paper or in many cases they go "there is something I don't quite get here" and that leads to lots of comments from people in the field and I think this is where we are evolving to... away from the traditional peer review system. The publication is seen to be the start of the scientific process and not the end, where you generate debate.

My questions:

Are there other such examples in physics? Can you point me towards a blog/site or something of the sort that goes through these things?

Are there any blogs, articles or essays that discuss problems surrounding the peer-review process (in relation to physics). How it can improved, where alternative methods like post-publication review are discussed ? I know ChemBark for chemistry, I am looking for it's "equivalent" in physics.

migrated from physics.stackexchange.com Jul 17 '14 at 21:49

This question came from our site for active researchers, academics and students of physics.

  • 7
    You may also be interested in Retraction Watch, which is in principle multi-field, but tends to have biology and chemistry papers, mostly. – E.P. Jul 17 '14 at 16:19
  • 3
    The most (in)famous of these in physics is Jan-Hendrik Schon, who got ~10 papers through to Nature/Science until finally someone found several reused images. A rather enjoyable book, Plastic Fantastic, was written about the case. I second the recommendation of RetractionWatch, which I too keenly follow. I think that faking it in physics is less common as you often need to back your claims theoretically. That said, it is very easy to find faulty data due to incompetence (without malicious intent, just the pressure of publish or perish). Most don't want to publicly ridicule such research. – alarge Jul 18 '14 at 14:17
  • 3
    You can read only the physics category on Retraction Watch if you want: retractionwatch.com/category/by-subject/… – ff524 Jul 18 '14 at 14:48
  • 2
    @MarkJ That has nothing to do with physics, that's more about post-modernism and the obscurities, lack of intellectual rigor,etc., associated with it. It is very interesting nonetheless. – The very fluffy Panda Jul 19 '14 at 15:47
  • 1
    @MarkJ "problems surrounding peer review IN PHYSICS" – The very fluffy Panda Jul 19 '14 at 18:48
1

The methods of scientific misconduct in physics are not different from what you see on chembark (photoshopping, data creation and so on). This years nobel prize in chemisty went to a physicist as it happened often. The methodogical overlap is huge, mostly the topical focus is different, collaboration is frequent as interdiffusion of scholars between both branches. Also physics is such a broad field that the role of peer-review in different subfields has different importance.

Jan Hendrik Schön was an exceptional case, from which one can deduce the differences in revealing scientific misconduct and the role of peer review in physics vs. other scientific branches. There are not much cases like Jan Hendrik Schön in physics, as we have a very strong post-publication review system in physics/chemistry many scientific branches lack: the industry and the mathematical and experimental interdependence and overlap of most theories in neighboring subfields. It's stronger than the peer-rewiew system at the journal level and one of the reasons why physics is called a "hard science". This is really the crucial difference to other scientific branches and the reason why you will not find much important scientific misconduct by physicists in top journals. Hiring of top researchers by industry and patents are a strong incentive to avoid scientific misconduct. And top research in physics is correct, reproducable and valuable research, in branches like psychology or philosophy you really have problems to identify and judge good research by clear criterions. This is due to the nature of the topics. Jan Hendrik Schöns research career was finished, he will never do fundamental research again or wright a paper. He probably knew that is was just a matter of time considering the importance of his results that someone in academia or industry would reveal his misconduct, probably career obsessed and blind.

The case of Schön started in the physics community an open-ended discussion wether peer-review at the level of the journal is the place to detect such cases, there were not much widespread cases like this before (I can't remember). From the discussions I had it seems most physicists agreed that not peer review at the journal level was the problem in this case, but lack of review of his collaborators before submission. Unfortunately in the aftermath he was the only one sentenced and losing his PhD 2013. Take a look how much co-authors he had on most papers. Schön's other collaborators were cleared of all wrongdoing by an committee appointed by Bell Labs. They are nowadays professors at ETH Zürich and Stanford. They could not identify the fraudulent data or would have to invest to much effort?

To this it boils down, how much effort do you have to invest in peer review and what is the status quo of trust and mistrust in your subfield? Is mistrust prevailing? For some branches the status quo is problematic. In physics we can tolerate such a case of imperfect peer review at the journal level from time to time due to much stronger other inherent mechanisms other scientific branches lack. The subfield of priming in psychology for instance had to undergo rather exceptional processes to regain credibility and will have to do so on.

Of course, also other branches develop technological relevant ideas, prototypes, patents and are funded by industry, but the crucial point is that often published results in physics journals turn into an product within a decade. This time scale is important, as it has strong implications on the quality of work of experimental physicists as the case of Jan Hendrik Schön has shown. In other scientific branches, e.g. psychology, you nearly never have to fear this kind of very strong post-publication review (where experimentalists in industry and academia will try to analyze and rebuild your experiment bit by bit, much stronger than any review by an editor or collaborators) or discovery-idea-patent-product will last several decades (e.g. drug development in medicine), so nearly as long as the career of most scholars and some decide to take this risk. In physics/chemistry, for widely studied and interesting research topics, your chance is zero, misconduct in the long term (which means 10, max. 20 years) is scientific kamikaze.

Please don't interpret my words as an underestimation of peer-review done at the journal level (in non-industry relevant fields it is the most important level), I'm a strong advocate of open access and open data and in my opinion the best hindrance of scientific misconduct would be if search engines like google scholar would directly link comments on arxiv to specific papers in the search results. If a bunch of co-authors decide to falsify data professionally (not like a school kid in the picture above), then only other research groups trying to reproduce this results can reveal the misconduct, post-publication review. In physics this will happen for sure. On retractionwatch you can read stories about journals that are not inclined to publish comments of other authors proving the bad quality of their peer-review. In physics comments have always been a vital part for public discussion of competing theoretical models and experimental data. But nowadays the publishers rely more on a more profit oriented model than decades ago. So central uncensored websites like pubpeer are really necessary from my point of view seeing a rise of plagiarism and misconduct, although their importance may vary strongly from branch to branch. Physics, in my opinion, is not in trouble, actually the time scale described above has become even shorter since the rise of the internet due to better and faster access to software, data and knowledge. Unfortunately in humanities the phenomenon of "copy, shake, paste" vice versa seems to have risen.

  • See the comment at my answer below. – just-learning Nov 4 '14 at 17:18
  • hmm interesting! sorry for the late response.. seems u didnt get the bounty... – The very fluffy Panda Nov 7 '14 at 11:58
3
+250

I would like to slightly disagree with the Hauser's answer regarding the possible methods of scientific misconduct in physics.

Unfortunately, in more theoretical parts of physics some people can manage to pretend that some complete and utter nonsense represents a new result well enough to fool the referees and editors and thus beat the peer review system.

A prominent example of this (to our luck, very rare, at least AFAIK) phenomenon is the Bogdanov affair, see also e.g. the web page by John Baez on this and the Nature article on the ensuing scandal; a sample of the works involved is available through this Google Scholar query. The Bogdanov (a.k.a. Bogdanoff) brothers got their Ph.D.s using the papers in question, one in physics and one in mathematics, and in early 2000s have published a number of strange -- to put it very mildly -- papers in a number of reputed physics journals, including Classical and Quantum Gravity and Annals of Physics (and e.g. the Annals of Physics paper is still there -- it is not withdrawn until today). The papers were not just wrong -- they turned out to be downright gibberish.

In the light of the Bogdanov affair it is not a big surprise that sometimes peer review also fails to detect even a really large-scale plagiarism, as in the infamous case of more than 60 works by a group of Turkish physicists who plagiarized earlier work, see e.g. the Nature article and this post at Peter Woit's blog for details.

Unlike the Jan Hendrik Schön case, AFAIK the papers by Bogdanov brothers did not involve experimental data or predict outcomes of reasonably doable experiments, so their case is pretty much a pure peer review failure at the level of the referees and editors (and of course a failure of their thesis advisors too).

IMHO, the Bogdanov affair shows that the argument regarding going to production and post-publication review from the Hauser's answer does not seem to work well in the more theoretical parts of physics, especially when the authors make no readily verifiable experimental predictions.

  • 2
    In any scientific branch you can "beat the peer review system", but the point is, in physics it doesn't stay undetected very long, and so you loose your scientific career. And that's why it doesn't happen very often. Even the purely speculative stuff of this twins no experimentalist being interested in got detected and debunked. Now deducing and claiming from this that the mechanisms I described don't work for >90% of theoretical physics because of this single case is the utterly wrong conclusion. I think you either misread my answer or didn't read it completely. The collaboration of theorists – Hauser Nov 4 '14 at 14:04
  • 1
    Please answer the question (it's how review works in physics) and try to see the whole picture in this branch, what you say applies for every other scientific branch, where much more articles are poor and uninteresting and you could find dozens of such cases still doing research. This "affair" was over after few years. Show me other cases similar to Bogdanov/Schön which were debunked after 20 years in theoretical-experimental entangled physics and don't base a suspicion on – Hauser Nov 4 '14 at 19:28
  • 1
    system? Serious? Every kid knows that there cannot be a perfect peer review system at the journal level preventing cases like this in any scientific branch, even physics has corrupt journal editors and poor reviewers, nonetheless you have to explain why this branch has so much success, progress and scientific knowledge compared to branches with mediocre peer-rewiew and not post-publication review at all. You make a mountain out of molehill discrediting this whole field with a suspicion, not being even wrong for false reasoning although I gave many arguments and links which you seemingly didn't – Hauser Nov 4 '14 at 19:33
  • 1
    read. In other STEM branches over 70% of results cannot even be reproduced, in computer science people have written bots to fabricate articles and they were accepted... How does this compare to physics and why we don't have such problems in this branch? – Hauser Nov 4 '14 at 19:36
  • 1
    The OP asked for examples of problems around peer review in physics, and I provided such an example which exactly answers the OP's question, that's pretty much it :) – just-learning Nov 4 '14 at 19:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.