Correcting Preregistered Methods After the Fact

Question

I'm often asked to help conduct statistical analyses after a paper is rejected for statistical errors. In some cases, the study, including the methods and analyses, is preregistered in some way (e.g. the study that I'm currently dealing with was preregistered on clinicaltrials.gov). The problem that I face is that the preregistered statistical analyses are often incorrect and I would consider it a sort of malpractice on my part to perform the wrong analyses - even if that means deviating from the preregistered methods. Are there any guidelines regarding the academic/publishing ethics of deviating from a preregistered protocol?

Answer 1

As a fellow statistician who assists non-statistician researchers, I sympathise with this predicament. I actually gave a faculty talk at my university about a month ago, giving advice to researchers on writing methodological applications such as pre-registered descriptions of experiments. One of the things I stressed in that talk was the importance of describing statistical methods without "locking yourself in" to models or tests that end up being inappropriate for the data. Generally speaking, the statistical description in the pre-registered methodology should set out the general model form expected to be used, and give a description of the hypotheses that will be tested, but it should always give allowance for the researcher to make changes in the model when this is necessary to accommodate the data (e.g., when diagnostic plots show heteroskedasticity or non-independence of residuals).

Obviously all this is too late in the case you are dealing with, but it is a useful opportunity to raise those issues with the researchers who wrote the pre-registration document, and give some advice for future practice. It is also worth bearing in mind that the primary purpose of pre-registration of experiments is to prevent researchers from making post hoc comparisons and then passing them off as the hypothesis of initial interest. Pre-registration does not exist primarily to lock statisticians into model assumptions. In any case, logically, there are three possible ways you could proceed in this case:

• Good analysis trumps pre-registration: Do the correct statistical analysis, making whatever changes you need, even if this is contrary to the method/model set out in the pre-registration document. Make sure the write-up of the analysis and subsequent academic paper clearly disclose (and explain) any deviation from the pre-registered method. The reader will then be able to decide whether or not they still trust that the pre-registration has achieved the desired purpose.

• Pre-registration trumps good analysis: Do the analysis in accordance with the pre-registration document. Make sure the write-up of the analysis and subsequent academic paper clearly discloses the flaws in the analysis, but notes that they were fixed by the pre-registration decisions. Don't try to sweep these problems under the rug.

• Refuse to do the work: The last option is to refuse to do the statistical work on this project, on the basis that you cannot comply with the pre-registration method, while still doing good statistical analysis. This option leaves the other researchers in a bit of a bind, but some might say that that is the proper penalty for having locked in bad methodological decisions in the pre-registration document. (And even if you choose this option, some other statistician might take up the work, so they will still need to read this answer for advice!)

In my opinion, the first of these options is clearly superior to the second. You get a good analysis that is done properly, and the reader is able to decide if the pre-registration has achieved its purpose (usually to prevent post hoc comparisons) despite the changes. If you decide to go ahead with the analysis, I would suggest that you adopt this method, but make sure the other researchers agree that they will clearly disclose the changes you make to the method in the final paper (so that the reader knows that there was a deviation from the pre-registered method).