Peer review prior to beginning a study?

Question

In scientific research there are a number of problems today:

• Studies that aren't reproducible.
• Studies and papers that are criticized for their methods, conclusions, or bias.
• Industry-funded studies that are discounted due to accusation of conflict of interest.
• Individuals and groups who are defensive of their work due to their personal and/or financial investment, and are inherently not open-minded toward critical feedback.
• Even if researchers are open to critical feedback and scrutiny, it may still be infeasible to re-run a flawed study due to logistics, financial, and time limitations.

From my experience in engineering, arguably the most valuable ROI effort in a large project is getting early peer review before significant investment is made toward an approach.

If a group of peer reviewers had a chance to give their feedback on how a study was going to be performed, then the results are more likely to be valid, and the reviewers are more likely to accept whatever the results will be.

I am not a reseacher and am completely unfamiliar with the process of publishing a study or a paper. However, the only peer review I have heard of is where only final results are reviewed. I am curious:

1. Do journals, institutions, etc. have processes that require peer-review of initial proposals and procedures before a research study is allowed to commence?
2. If not, why not?

I suspect that such a process would have the potential to greatly increase the quality of studies, and prevent much wasted effort where studies are published that never gain acceptance (presumably due to quality problems but perhaps for non-technical reasons as well). Curious to hear others' thoughts.

Answer 1

Yes, some 225 journals do operate under such a model, called Registered Reports. As you suggest, the approach has a lot of advantages. The tagline is:

Registered Reports: Peer review before results are known to align scientific values and practices.

Clearly, this is not yet a majority view, but as a methodology it is in the ascendant. The umbrella term is preregistration.

Answer 2

Principles of academic freedom usually leave it up to the individual researcher to decide what to study (within ethical limits). They may certainly be judged (for promotion, tenure, raises, etc) based on whether their research is successful, but there's no "prior restraint" as it were. So an institution usually won't require this sort of pre-review. It is certainly wise for the researcher to seek advice regarding the feasibility of their planned study, but if they don't think it's worthwhile, or want to go ahead even if the reviewers think it's a bad idea, that is their prerogative (and their time wasted if it fails).

On the other hand, if they need money to do their work, there are certainly elaborate peer review processes for grant proposals. It's considered fine for a researcher to risk wasting their own time with a poorly planned study, but not for them to risk someone else's money.

For studies with possible ethical concerns (e.g. human or animal subjects), there will be an ethics review required before the work begins. This will mainly be aimed at whether the techniques are ethical instead of whether the overall study is likely to succeed, but one consideration may be whether the study's benefits outweigh potential risks to the subjects.

Answer 3

While only a minority of scientific work (I imagine mostly medical science?) is done under quite the conditions that you describe, there are two other things that help:

1. When funding organisations decide who to award grants to, the grant proposals are usually subject to peer review. I imagine that this is, in effect, quite similar to what you were thinking of.

2. IN some areas it's common for large projects to have a steering group of other academics and/or representatives from industry that the PI reports to every so often, and which will offer advice. It probably isn't peer review in quite the way that you were thinking, but it does serve many of the same functions.