Changing methods/ methodologies mid-project

Question

Is changing research methods/ methodologies in the middle of running a research project a good thing? Are there any best practices for doing so? Links to projects that document a methodological shift mid-project would be much appreciated.

Answer 1

Do you keep a methodology notebook? If not, it is high time you start now.

It is important in the experimental sciences to always document exactly how the experiment is performed, and it is often recommended to keep a methodology notebook aside from your data notebook. All current and past/obsolete methodologies should be documented and dated (so it is clear exactly when your lab switched to a new methodology). All data on experiments should include reference to the correct methodology. (A lab wiki may be a good way to manage this.) For a bit more discussion you can see this guideline from the neurosurgery department of UCSF.

Changes in methodology may or may not be a good thing. A change of methodology obviously introduces a new variable into your research. And depending on the research outcome, the data pre and post methodology change may or may not be compatible, and it may or may not be ethical to present the data as one single dataset. So if your experiment only has limited number of runs for which systematic bias that may be introduced by the new methodology cannot be statistically ruled out, you may be better off not switching. (Unless, of course, you are in the biomedical sciences where sometimes methodological shifts are compulsory, even though it means essentially aborting the experiment.)

Methodological shifts are not forbidden per se, as long as all the methodology are clearly documented. The issue is more with how you interpret the data afterwards: it maybe necessary to present your data as two separate experiments instead of one single experiment. (Depending on your field there are different culturally accepted level of "fudging" when it comes to the methodology section of a paper; you should always have an argument handy as to why changing the methodology does not really change the experiment.) As long as each data point can be clearly referenced to the method that was used to obtain that data point, your experiment is reproducible and ethically you've covered your behind.

(This, of course, does not address the bureaucratic issue of whether your funding agency and such would frown on deviating from the funding proposal, and whether your research institution have specific guidelines concerning methodology shifts.)