Is it appropriate to describe our methodological mistakes in manuscript?

Question

I work outside academia, but am working on a project with a master's student. During data collection, we had some skip-logic errors pop up while administering the survey online and it resulted in some superfluous data. She and her adviser decided to throw out all affected data and only analyze open-field text, and outlined these mistakes in great detail on her master's paper. While writing our joint manuscript for publication, is it appropriate to outline these programming errors?

Answer 1

It would be inappropriate not mention the mishaps in my opinion. But there is no need to document this in great detail, as you mentioned. I would comment during the revision process that you feel the description of the "skip-logic section" is too wordy, and it should be trimmed to be more concise.

Keep the focus on the positive findings as opposed to mishaps.

Answer 2

I would also mention significant failures in the methodology because it can serve as an important warning for future researchers. For instance, needing to change the number of samples or the testing period because of experimental (or computational) difficulties may be necessary, and it would be helpful for others to know why you needed to change the protocol. In addition, reporting such issues is important for purposes of reproducibility.

However, as the others have posted, there is absolutely no need to spend huge amounts of space discussing such issues, unless your paper is attempting to correct the methodological mistakes of others in the literature.

Answer 3

In the Methods section we detail the process of obtaining data to analysis. Errors resulting in loss of data is a deviation from the said protocol and hence should be reported. The details need not be excruciating fine; a sentence to explain the skip logic pattern and that affected subsections were not analyzed should be sufficient. This is by no means an uncommon practice: there are generic missing, loss of data storage media, biological samples expired, or even contamination in lab. If any of these activities can potentially bias your end result or lower your statistical power, then they should be reported.

Judging from what you said that the student and the supervisor had to throw away all multiple choices and only kept open-ended question, I am guessing the skip pattern problem was extensive? If that's the case, you should try to gain full understanding of the error and carefully decide if any of the data are actually salvageable. For instance, if the skip pattern problem had made the survey questions appear to be illogical, would you still expect the respondents to be in their right mind when filling in the open-ended questions?

The master's paper needs to happen probably because the student has done the due diligence and finished a project, despite with technical errors. But data mishap that is sufficient for a master's thesis does not imply the data are fit for publication. As your name will be on it, if you feel uncomfortable, request to be removed from the author list. You may opt for acknowledgement or even no relationship.