13

Could removing outliers be called cherrypicking data? Because, to run some tests, outliers have to be removed. And they even turn up unexpectedly ruining good data.

I still haven't yet started acquiring data for my project. Still collecting samples. I've heard a senior say that he removes them and I really doubted if such a thing could be done. Also, I've read that ANOVAs work on the assumption that outliers shouldn't be there. (correct me if wrong)

closed as off-topic by Scott Seidman, Daniel R. Collins, Cape Code, gerrit, EnergyNumbers Aug 16 '16 at 14:09

  • This question does not appear to be about academia within the scope defined in the help center.
If this question can be reworded to fit the rules in the help center, please edit the question.

  • 8
    Do you have an idea why the outliers are outliers? Poorly calibrated instrument, features that defy common sense, entry error? – user60356 Aug 15 '16 at 20:04
  • 2
    Also, I'm not aware of any statistical tests that fail when they have outliers, the results might not be accurate. What tests do you have in mind? – user60356 Aug 15 '16 at 20:18
  • 8
    Also note that if you expect a lot of outliers, you can use tests which are more robust towards outliers, for example use a rank based test rather than a t-test. Then you don't have to remove the points. – air Aug 15 '16 at 21:07
  • 8
    I'm voting to close this question as off-topic because this is a statistics question suitable for cross-validation, but does not offer enough context and detail to support a meaningful answer – Scott Seidman Aug 15 '16 at 22:13
  • 3
    If you do remove outliers, be sure to very clearly indicate that you have done so and what precisely your criteria for what counted as an outlier were in any paper that uses the data. If you do so, you may be pinged for poor analytical technique, but I doubt you could be accused of being dishonest. – Kevin - Reinstate Monica Aug 16 '16 at 6:00
23

One major pitfall is going into analyzing the data without an a priori protocol to deal with outliers. Time to time, tension rises between analysts and investigators on whether a point should be removed or not. Investigator may wish to keep it because it drives the significant result; analyst may be anxious to keep it because, well, it drives the significant result. Much can be avoided if there is a agreed-upon standard.

Anyhow, here are some suggestions (slightly scrabbled):

  • Reconcile if it's an instrument or data entry error.
  • If it's a plausible value but behaving like an outlier, a few methods exist:
    • Do not bury plausible extreme values in favor of showing the after-trimming data only.
    • Consider reporting with and without it, discuss the differences.
    • Use statistics that are more tolerant to extreme values. For instance: Non-parametric statistics or robust estimation methods.
  • Keep all written record about related changes, make sure date, time, person, decision made, etc. are all recorded.
  • If correction needs to be made, do not change the raw data. Do so using script/syntax and save that syntax for reference.
  • Explain all outlier-related data change in the report/manuscript. Reasons and methods should be provided so that your analysis can be replicated.
  • If there isn't a protocol, set one up now before more instances like this came up.
  • Do not identify outlier after data analysis results are known. If possible, discuss with your colleagues about the outlier without telling them the hypothesis test results.
  • +1 for the introduction. Starting to perform mathematical operations without knowing what the data represents is just asking for trouble. For example, having a function about a person which returns one value if the person is the Pope, and another value if the person is not the Pope, guess what will happen if a naive approach is taken. – vsz Aug 22 '16 at 5:10
13

The questions in your title and body are different in a rather significant way.

In answer to "Could removing outliers be called cherrypicking data?": yes, of course it could.

In answer to "Would it be called cherrypicking data?", that depends on your justification (and apparent motivation).

To be honest I would generally be very suspicious of any paper which removed data points without a very clear and justifiable reason. It isn't possible to tell whether that's the case for your example, as you haven't given any information on it, and additionally for statistical advice it would be more appropriate to ask this over at Cross-Validated.

There are statistical methods for identifying how much of an outlier a point is (leverage, for example) but they are not in and of themselves sufficient justification for removing a point.

[edit] thanks for adding more information to your question. Firstly, it's great that you haven't started collecting data yet, and well done for thinking about this at this stage. As @penguin-knight suggests, formulate a protocol for data collection and identifying suspect data before you start (a priori). However, do remember that the only defensible reason for removing an outlying data point is because you believe it is inaccurate and reflects a problem with how that sample was generated or quantified.

Secondly, regarding the ANOVA part of your question: one of the assumptions when using an ANOVA is that your residuals are normally distributed. 'Outliers' may indicate that this assumption is incorrect. If this is the case, simply ignoring some points is not the correct way to deal with this. Your simplest options are:

1

Display (somewhere) all the data. Explain why each deleted point is removed; one idea may be "known bad instrument setting," another is "exceeded xx.xx standard deviations" or the like. This allows a critic to decide if you are in error and where but does not allow said critic to claim you desk-drawered unwanted results.

1

You could simply include all data in a set of results, and then present a second "filtered" result set excluding the outliers. This seems like the most honest and comprehensive way to deal with this.

Not the answer you're looking for? Browse other questions tagged or ask your own question.