Is peer review process totally theoretical?

Question

Does the peer review process involve only theoretical analysis or it contains some practical components?

Answer 1

There are many reasons that peer review generally "works" but isn't perfect. In some fields, such as mathematics, a paper is normally expected to contain a sufficiently detailed proof of any claims (theorems) that a specialist can check the validity.

In other fields, this isn't possible within the time frame or the resources available within the review process. As your first version of the question suggested, the best way to verify claims made using, for example experimental or statistical methods, is to replicate the study independently. This takes both time and money and so is infeasible if results are to be made available in a timely manner.

But there are several safeguards that tend to weed out the garbage, though they don't eliminate it with 100% certainty. While an author may not know who the reviewers of a given paper are, reputable journals use reputable and experienced reviewers. Typically a paper is reviewed by more than one person and most will have some expertise in the specific subject matter. They will know what is reasonable to expect and what is not. A reviewer will first ask whether any claim is at all reasonable. Unreasonable claims either get rejected or get extra scrutiny.

Reputation in general is a check on bad results, though, again, not absolute. If an author is associated with a reputable institution, they want to protect not only their personal reputation but that of their institution. A reviewer seeing a paper from an independent researcher, associated with no reputable institution, won't have the automatic confidence that the study was carried out properly and so might reject it, or give it extra scrutiny. But many such papers don't pass the "smell test" and get quickly rejected.

There is, in fact, a "web of trust" in the research community. An assumption that people, generally, do honest work and try to avoid error. An assumption that they check their work and try hard not to publish things that later turn out to be wrong.

For an academic wanting to build a reputation, there is very strong incentive to get it right. It is one thing to have a paper rejected by a good journal and quite another if it is learned later that there was intentional misconduct, even sloppiness. An good academic reputation is easier to break than it is to build. And much harder to rebuild.

Also note, that incorrect science is sometimes published even when everything is done correctly and ethically. Unseen factors and even statistical anomalies do occur in work even when it is done carefully. There is no absolute guarantee in an imperfect world.

But, yes, it is possible to fake data. It is possible to assert that procedures were followed when they were not. But people doing that get a reputation that will follow them.