10

I am a PhD student, and my supervisors have a "different approach" for doing research. They do not dig into the literature to find a gap and then do research on that. Instead, they just do some surveys and then hand the data over to their students, who go through it in order to find an interesting research question to address. Is this fishing for data? Is there a body in which this could be anonymously reported? Or is just one of the bad practices that one can sadly find in academia, and unfortunately in this case on my PhD research?

  • 7
    This is not necessarily bad, as long as the hypothesis that a student may formulate after viewing the data is tested with subsequent experiments to do exactly that. All research starts with a (possibly surprising) observation. – louic Dec 11 '18 at 14:09
  • 3
    Maybe you need to say more. What you write sounds to me as legitimate. "What is still unknown that is worth exploring?" Are you upset that they are leaving this to students to do? That sounds like valuable training. I don't see the problem. To do research you first need to know what are the unknowns. – Buffy Dec 11 '18 at 14:14
12

"Fishing" is fine for exploratory analysis: formulating hypotheses to be tested. It can even be publishable if you are straightforward with your fishing: reporting data is not unethical. Fishing without accounting for your fishing, however, is quite problematic, as your question implies.

Unfortunately, in some fields it is common to go straight to publishing exploratory analyses as if they were not exploratory. Statistically, this is a bad idea; the 'replication crisis' in especially psychology is an example of the consequences. If you torture the data enough, nature will always confess.

As awareness of this problem spreads, some journals and funding agencies have now gotten very strict, for example with clinical trials, and demand that trials be preregistered to be published/funded to avoid the increased rate of false-positives from "fishing" through data. I'm not aware of anything similar in other types of research, but certainly that's the direction that statisticians would suggest to avoid reporting misleading results.

As far as what you can do, it really depends on the standards of the field, and you are in a vulnerable position as a student. I would recommend you not publish "fished" data with your own name on it unless the extent of the exploratory analysis is made clear, and that you advocate to apply appropriate statistical approaches to fished data, like using those results to inform a subsequent study to properly control your false positive rate.

  • 2
    +1 The keyword here is "exploratory research". Exploratory research isn't in any way bad or unethical, as long as it's correctly labelled (and proper procedure on data collecting and reporting are followed). I feel exploratory research has to some extent been a collateral damage in the current wave of reproducibility angst. – xLeitix Dec 12 '18 at 12:05
7

In statistical analysis we draw a distinction between two types of data analysis, that depend on whether you already have a well-formulated research hypothesis you want to test, or you are merely looking for interesting hypotheses to test. Statisticians are also cognisant of the dangers of testing bias in cases when researchers are given flexibility in identifying hypotheses of interest, or making model choices, after already seeing the data they are using. I would strongly suggest you read some of the statistical literature about these issues, and have a look at some of the discussion by Andrew Gelman on researcher degrees-of-freedom (see e.g., Gelman and Loman 2013). Here is a basic summary:

  • Exploratory Data Analysis: In this form of analysis you do not have a research question formulated (or your research question is not yet formulated with sufficient precision to be testable) and you are exploring a data set to look at patterns and formulate possible research questions. This analysis generally consists of graphical methods and model-fitting that is done with a view to summarising trends, rather than for formal testing. Relationships that emerge from statistical analysis in this phase are treated only as research hypotheses for the future. Principles and procedures for this kind of data analysis can be found in many statistical papers and books (e.g., Behrens 1997).

  • Confirmatory Data Analysis: In this form of analysis you already have a well-formulated testable hypothesis prior to seeing your data. This analysis generally involves model-fitting from a pre-specified class of models, and formal statistical testing of pre-specified hypotheses. For added rigour, researchers often pre-register the details of the analysis and hypotheses, to ensure that this choice is not affected by the data. Generally researchers make an effort here to minimise their flexibility, so as to ensure that they are not making testing decisions that are biased by seeing the data. If done correctly, formal statistical testing of pre-specified hypotheses establishes unbiased evidence about those hypotheses.

  • IMPORTANT - Don't use the same data for both! If you use the same data set for EDA and CDA, this biases your tests strongly towards confirmation of the hypotheses identified in the exploratory stage. The reason for this is that the choice of hypotheses or testing methods may be affected by seeing the data. In the case of deliberate "fishing" the hypotheses of interest are chosen directly from the fact that strong evidence for them is observed in the data. Even in the absence of deliberate fishing for hypotheses of this kind, there can also be bias if researchers have too much flexibility to determine their hypotheses or modelling choices after seeing the data.

From your description it sounds like you are being given data for the purposes of performing exploratory data analysis. There is nothing inherently wrong with this. In this context it is legitimate to "go fishing" for patterns and trends, so long as the identified patterns and trends are treated as hypotheses for future testing, rather than conclusions from the data. The important thing here is not to use that same data to perform formal testing of the hypothesised pattern or trend (since this will be heavily biased towards confirmation). What should be done instead is to then collect new data with a well-formulated testable hypothesis in mind, and use that new data for formal testing.

Both exploratory and confirmatory data analysis are legitimate forms of research if done correctly. There is no in-principle reason that pure exploratory analysis could not be published, and indeed, it would be wonderful to see more of this type of analysis get published in academic journals. Unfortunately these issues are not always well understood, and the pressure to publish "statistically significant" tends to induce a bias against publishing exploratory analysis, and sometimes leads researchers either to ignore EDA, or try to shoehorn it incorrectly into CDA.

  • Would it be legitimate to split an available data set in two data sets, one for exploratory analysis, and one for confirmatory analysis of the hypotheses derived from exploratory analysis? – lighthouse keeper Dec 14 '18 at 8:58
  • 1
    Yes, absolutely! Indeed, that is a good way to do it so as to avoid having to do data-collection twice. The best method for this is to choose the desired size of your two groups and then use simple-random-sampling to split them out into disjoint groups of the required sizes. (Make sure you don't look at your CDA data while doing EDA.) – Ben Dec 14 '18 at 9:13

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.