Academia Stack Exchange is a question and answer site for academics and those enrolled in higher education. It only takes a minute to sign up.
Sign up to join this community
Could removing outliers be called cherrypicking data? Because, to run some tests, outliers have to be removed. And they even turn up unexpectedly ruining good data.
I still haven't yet started acquiring data for my project. Still collecting samples. I've heard a senior say that he removes them and I really doubted if such a thing could be done. Also, I've read that ANOVAs work on the assumption that outliers shouldn't be there. (correct me if wrong)
One major pitfall is going into analyzing the data without an a priori protocol to deal with outliers. Time to time, tension rises between analysts and investigators on whether a point should be removed or not. Investigator may wish to keep it because it drives the significant result; analyst may be anxious to keep it because, well, it drives the significant result. Much can be avoided if there is a agreed-upon standard.
Anyhow, here are some suggestions (slightly scrabbled):
The questions in your title and body are different in a rather significant way.
In answer to "Could removing outliers be called cherrypicking data?": yes, of course it could.
In answer to "Would it be called cherrypicking data?", that depends on your justification (and apparent motivation).
To be honest I would generally be very suspicious of any paper which removed data points without a very clear and justifiable reason. It isn't possible to tell whether that's the case for your example, as you haven't given any information on it, and additionally for statistical advice it would be more appropriate to ask this over at Cross-Validated.
There are statistical methods for identifying how much of an outlier a point is (leverage, for example) but they are not in and of themselves sufficient justification for removing a point.
[edit] thanks for adding more information to your question. Firstly, it's great that you haven't started collecting data yet, and well done for thinking about this at this stage. As @penguin-knight suggests, formulate a protocol for data collection and identifying suspect data before you start (a priori). However, do remember that the only defensible reason for removing an outlying data point is because you believe it is inaccurate and reflects a problem with how that sample was generated or quantified.
Secondly, regarding the ANOVA part of your question: one of the assumptions when using an ANOVA is that your residuals are normally distributed. 'Outliers' may indicate that this assumption is incorrect. If this is the case, simply ignoring some points is not the correct way to deal with this. Your simplest options are:
apply a transform to your data, for example a log-transform. There is a lengthier discussion of transforms and ANOVAs here: https://stats.stackexchange.com/questions/75005/non-normal-data-for-two-way-anova-which-transformation-to-choose
use a non-parametric test (which does not rely on the assumption of normally-distributed residuals) - such as the Mann-Whitney U test if you are comparing two groups.
Display (somewhere) all the data. Explain why each deleted point is removed; one idea may be "known bad instrument setting," another is "exceeded xx.xx standard deviations" or the like. This allows a critic to decide if you are in error and where but does not allow said critic to claim you desk-drawered unwanted results.
You could simply include all data in a set of results, and then present a second "filtered" result set excluding the outliers. This seems like the most honest and comprehensive way to deal with this.
