13

Why aren't all experimental results of any validity required to register the experimental plan used for the research paper in question, ahead of time, giving the understanding that the results will be made known. Why,

because

  1. The description of the experimental procedure should be untouchable once the experiment commences, and most importantly
  2. it removes the option of simply failing to publish. This option may introduce bias as there may be myriad reasons why a valid experimental result could be withheld. Such reasons include those given by the biases of the experimenters themselves or the judged impact of an out-of-the-box result.

By all appearances, if one is to embark on an experimental program, shouldn't one be bound to disclose the results? Registration seems to be the only way to enforce this. It wouldn't be taxing... simply emailing a pdf to a database system of some kind would be enough.

I'm guessing many papers undergo this process on an informal level, but I doubt every research paper has some sort of registration like this.

[My background is in mathematics, not the sciences proper.]

2
  • 6
    One thing to note is that the further one is from medicine, the less things resemble the structured "scientific method" as taught to us all in middle school. Cutting-edge science is necessarily largely exploratory; the sort of controlled hypothesis testing you're imagining is often left for followup investigations.
    – user4512
    Aug 4, 2016 at 17:49
  • From the perspective of biological sciences, some experiments are very straightforward and would work the way you describe. But for many of them it is very difficult to determine the success of an experiment beforehand and the protocol may evolve as time goes on. Even after pilot experiments there are still snags and dead ends that may force you to take the research in direction different than the original protocol. Then there are also issues of timing (e.g. age of animals) that may require protocol modification. It's just not realistic in many cases.
    – syntonicC
    Aug 5, 2016 at 3:53

1 Answer 1

34

Pre-registration is not the easy "cure" for publication bias that you seem to think it is. In short: Who is going to enforce that authors actually follow their pre-registered plan and publish their results (which takes time, effort, and money to check)? What kind of consequences can be imposed on those who don't?

There are no major incentives in place for editors, reviewers, or authors to enforce this, except for their desire to advance the field and enhance the rigor of their discipline. Clearly, this intrinsic desire is not enough to prevent authors from throwing out negative results, reviewers from recommending rejection of negative results for not being interesting, or editors from declining to publish negative results. Pre-registration alone does not change that incentive structure; changing the incentive structure is not easy to do.


In medicine, pre-registration of clinical trials has been required by many major journals as a prerequisite for consideration for publication since around 2004-2005. However, there is no clear evidence that this has had an effect on publication bias in that field. Authors continue to submit papers in which the primary outcome of experimental protocol deviates from the registration, and reviewers often don't check the registration document to catch this. Non-publication is still common. And many journals are unwilling to take a hard line on pre-registration.

You said,

The description of the experimental procedure should be untouchable once the experiment commences,

In medicine, the pre-registration did not actually achieve this. See e.g. Mathieu, S., Boutron, I., Moher, D., Altman, D.G., & Ravaud, P. (2009). Comparison of Registered and Published Primary Outcomes in Randomized Controlled Trials JAMA, 302 (9):

Of the 323 included trials, 147 (45.5%) were adequately registered (ie, registered before the end of the trial, with the primary outcome clearly specified). Trial registration was lacking for 89 published reports (27.6%), 45 trials (13.9%) were registered after the completion of the study, 39 (12%) were registered with no or an unclear description of the primary outcome, and 3 (0.9%) were registered after the completion of the study and had an unclear description of the primary outcome. Among articles with trials adequately registered, 31% (46 of 147) showed some evidence of discrepancies between the outcomes registered and the outcomes published. The influence of these discrepancies could be assessed in only half of them and in these statistically significant results were favored in 82.6% (19 of 23).

You also said,

it removes the option of simply failing to publish

Pre-registration does not, as you seem to believe, compel the experimenter to publish the results. How could it? Who would enforce this?

Pre-registration can potentially make it much easier for the results to be published than they otherwise would have been; some journals offer peer-review and conditionally accept the article based on the pre-registration, before the experiment has actually been conducted. (Negative results can be quite difficult to publish otherwise.)

But non-publication remains common in medicine, even among registered clinical trials. From Ross, J. S., Mulvey, G. K., Hines, E. M., Nissen, S. E., & Krumholz, H. M. (2009). Trial publication after registration in ClinicalTrials. Gov: a cross-sectional analysis. PLoS Med, 6(9), e1000144:

We examined reporting of registration information among a cross-section of trials that had been registered at ClinicalTrials.gov after December 31, 1999 and updated as having been completed by June 8, 2007, excluding phase I trials. We then determined publication status among a random 10% subsample by searching MEDLINE using a systematic protocol, after excluding trials completed after December 31, 2005 to allow at least 2 y for publication following completion. Among the full sample of completed trials (n = 7,515), nearly 100% reported all data elements mandated by ClinicalTrials.gov, such as intervention and sponsorship. Optional data element reporting varied, with 53% reporting trial end date, 66% reporting primary outcome, and 87% reporting trial start date. Among the 10% subsample, less than half (311 of 677, 46%) of trials were published, among which 96 (31%) provided a citation within ClinicalTrials.gov of a publication describing trial results. Trials primarily sponsored by industry (40%, 144 of 357) were less likely to be published when compared with nonindustry/nongovernment sponsored trials (56%, 110 of 198; p<0.001), but there was no significant difference when compared with government sponsored trials (47%, 57 of 122; p = 0.22). Among trials that reported an end date, 75 of 123 (61%) completed prior to 2004, 50 of 96 (52%) completed during 2004, and 62 of 149 (42%) completed during 2005 were published (p = 0.006).

And authors have many reasons for not publishing, reasons that don't necessarily go away with pre-registration. From Song, F., Loke, Y., & Hooper, L. (2014). Why are medical and health-related studies not being published? A systematic review of reasons given by investigators. PloS one, 9(10), e110418:

We included 54 survey reports. Data from 38 included reports were available to estimate proportions of at least one reason given for not publishing studies. The proportion of non-submission among unpublished studies ranged from 55% to 100%, with a median of 85%. The reasons given by investigators for not publishing their studies included: lack of time or low priority (median 33%), studies being incomplete (median 15%), study not for publication (median 14%), manuscript in preparation or under review (median 12%), unimportant or negative result (median 12%), poor study quality or design (median 11%), fear of rejection (median 12%), rejection by journals (median 6%), author or co-author problems (median 10%), and sponsor or funder problems (median 9%). In general, the frequency of reasons given for non-publication was not associated with the source of unpublished studies, study design, or time when a survey was conducted.

Actually verifying that the details of a manuscript match the pre-registration document involves some overhead for reviewers and editors. See e.g. Mathieu, S., Chan, A. W., & Ravaud, P. (2013). Use of trial register information during the peer review process. PLoS One, 8(4), e59910, which found:

1,136 authors (n = 713) and reviewers (n = 423) responded (37.5%); 676 (59.5%) had reviewed an article reporting a clinical trial in the past 2 years. Among these, 232 (34.3%) examined information registered on a trial registry. If one or more items (primary outcome, eligibility criteria, etc.) differed between the registry record and the manuscript, 206 (88.8%) mentioned the discrepancy in their review comments, 46 (19.8%) advised editors not to accept the manuscript, and 8 did nothing. The reviewers' reasons for not using the trial registry information included a lack of registration number in the manuscript (n = 132; 34.2%), lack of time (n = 128; 33.2%), lack of usefulness of registered information for peer review (n = 100; 25.9%), lack of awareness about registries (n = 54; 14%), and excessive complexity of the process (n = 39; 10.1%).

Finally, publishers and editors may be reluctant to require pre-registration because they fear that authors will simply go elsewhere. From Wager, E., & Williams, P. (2013). “Hardly worth the effort”? Medical journals’ policies and their editors’ and publishers’ views on trial registration and publication bias: quantitative and qualitative study:

Only 55/200 journals (28%) required trial registration according to their instructions and a further three (2%) encouraged it. The editors and publishers interviewed explained their journals’ reluctance to require registration in terms of not wanting to lose out to rival journals, not wanting to reject otherwise sound articles or submissions from developing countries, and perceptions that such policies were not relevant to all journals. Some interviewees considered that registration was unnecessary for small or exploratory studies.

4
  • 1
    This is an excellent answer. Aug 4, 2016 at 8:21
  • 3
    I'm not sure I've ever done an experiment where what we did that was written up in papers was actually what we thought we would do, except in broad enough terms as to make a 'pre-registration' useless. When trying to do something that no one else has done, you don't really know what you are precisely going to do.
    – Jon Custer
    Aug 4, 2016 at 14:55
  • Obviously authors can't be prevented from abandoning pre-registered experiments that go badly; what pre-registration could usefully do, however, is adjust the thresholds of statistical significance. If someone pre-registers an experiment unlike anything they've done before, the sample size for a statistically-significant result should be smaller than if the experiment is similar to twenty others by the same person that have been pre-registered but not yielded a publishable positive result.
    – supercat
    Aug 4, 2016 at 16:29
  • 2
    Pre-registration should not be required for exploratory tests whose outcome will never be published as being meaningful evidence of anything beyond justifying that it may be worthwhile to conduct further, pre-registered, experiments.
    – supercat
    Aug 4, 2016 at 16:34

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .