57

I have a scatter(log-)plot with some 10,000 data points that plots the running time of some algorithm against the input on random instances of some problem.

I have a lot of these plots, and due to space constraints I can't exactly dedicate a large amount of space to them. The plots are 1-2 inches tall. The trouble is that a handful (maybe 5 or so) of these samples have taken a ridiculously short amount of time to complete (say, a few milliseconds), whereas pretty much all of the other data points have taken 2-3 orders of magnitude longer.

I'm trying to show that my algorithm is fast, so I figure it shouldn't hurt to just omit these handful of data points and generate more samples, right?

I feel like mentioning anything would unnecessarily confuse the reader, and keeping them would annoy the reader since the plot would have a large amount of blank space. And obviously it's not like I'm trying to suppress evidence against my research or something---the discarded data is only in favor of my algorithm.

Would I be violating some ethical code here if I just discard those samples without mentioning anything so that my plots look nicer? Is it unscientific? And if so would anyone actually care?

Improve this question
7
  • Did you mean 5 out of 10,000 were super faster?
    – Nobody
    Commented Aug 30, 2015 at 12:29
  • @scaaahu: Yeah. (Is there another possible interpretation of my question that I'm missing?)
    – user541686
    Commented Aug 30, 2015 at 12:30
  • 2
    No, I just want to make sure I read your question right. Five out of 10,000 is extra-ordinary. There could be some other reasons for that to happen.
    – Nobody
    Commented Aug 30, 2015 at 12:40
  • 11
    Would a broken axis solve your problem?
    – Wrzlprmft ♦
    Commented Aug 30, 2015 at 15:44
  • 50
    Is it unethical/unscientific to omit outlier data — Yes.
    – JeffE
    Commented Aug 30, 2015 at 18:48

8 Answers 8

Reset to default
132

Science aims at revelation and insight. Before you can even consider dropping these samples, you need to understand why they exist.

The reason is that unexpectedly "good" data can be just as much a sign of problems with your theory as unexpectedly bad data. Are these data points telling you that you've got a bug in your algorithm? Are they saying that the instrumentation you were using for timing was unreliable or not calibrated properly? Or is it just that under certain circumstances your random problems happen to be exceedingly easy to solve? There may be other possibilities as well.

If you can't determine why the outliers exist, then you must include them, in order to help the reader evaluate your work. If you determine they exist for a problematic reason, then, well, you're not ready to publish yet. If you determine they exist for a benign reason, then you can drop them from the figure, but you must explain exactly what you've done in the text and why, or else you risk misleading the reader.

At the end of the day, the data you got is the data you got, and you need to deal with it honestly.

Improve this answer
4
  • 46
    ... problems with your theory or with the experiment. Remember, "I can't explain that" is where the real discoveries are found.
    – keshlam
    Commented Aug 30, 2015 at 14:52
  • 6
    At the very least, I think you need to find out if the outlier values are reproducible by re-running the same input - even if the input was generated "randomly", you can presumably save it and re-use it. If they are not reproducible and you can't explain why, the next question to ask could well be "do any of your results demonstrate anything at all".
    – alephzero
    Commented Aug 30, 2015 at 17:14
  • 1
    Also: there have been several studies where what appeared to be outliers were, in fact, the valid data. I wish I could remember them off the top of my head, but I read something about it in the last year or so.
    – Wayne Werner
    Commented Aug 31, 2015 at 14:41
  • 12
    @keshlam: Big discoveries don't go "Eureka!" They go "Huh, that's funny."
    – Mason Wheeler
    Commented Aug 31, 2015 at 18:28
61

Is it unscientific?

Yes. The purpose of these plots is not to show that the algorithm is fast, but rather to give an accurate picture of its speed. Intentionally removing outliers without explanation gives a distorted picture. It's OK if you are clear about it (for example, explaining in the caption or text about the outliers that were removed and why), but not if you do it silently.

These outliers could really matter:

  1. Maybe they indicate a bug in your code, in which case removing them would look like you were deliberately covering up sloppy work.

  2. Explaining the outliers could be scientifically interesting and important (perhaps it could lead to an even faster algorithm), but nobody will try if they don't even know they are there.

  3. If someone else implements the algorithm and compares their results to yours, they may waste time trying to understand why they have outliers and you don't.

Improve this answer
4
  • 4
    Especially for algorithms, where the performance of edge cases is important, the notion that you can toss out outliers is...a little problematic.
    – Fomite
    Commented Aug 30, 2015 at 23:11
  • @Fomite: Well my idea was that worst cases are almost always interesting, but best cases seldom are, since it's easy to make an algorithm have a "fast path" for returning answers to easy queries. For example, imagine if you were trying to sort a list, only to find that the list was already sorted. Then you wouldn't need to do anything else at all, and algorithm would finish much faster in these cases. But (unless your algorithm is recursive) this best-case running time would be uninteresting when benchmarking the performance of your sorting algorithm, so you might try to exclude it.
    – user541686
    Commented Aug 31, 2015 at 18:50
  • @Mehrdad it can be useful to have such an algorithm in a situation where you expect your lists to be ordered most of the time, but not always. Best cases are indeed interesting, and as your own testing suggest, not so contrived (once every two thousands random cases).
    – Davidmh
    Commented Sep 2, 2015 at 10:59
  • @Davidmh: I never denied the usefulness of having such an algorithm. I denied the usefulness of plotting the best-case behavior of such an algorithm in many (not all) cases. It makes it harder to meaningfully compare the algorithm against others unless for some reason you expect the lists to be ordered most of the time, which you generally don't.
    – user541686
    Commented Sep 3, 2015 at 6:13
33

Yes. For a practical scenario, imagine someone trying to replicate your research, going only by your paper, and beating themselves over the head thinking they have a bug, because their plot shows these weird outliers.

Basically, you have to give people all the information, because you don't know how they're going to use your paper. Probably, most readers won't care about the outliers, and probably they would judge the method the same, but it's not up to you to make that decision.

Of course, you have to filter out the noise, somehow. Usually, the trick is to figure how to give the reader all the information, while allowing them to focus on what's important. In your case, I would just say in the caption to the plot, that 5 runs of the algorithm were so fast that they were outside the scale of the plot (or something to that effect).

Improve this answer
1
  • 4
    +1 for the first paragraph. That's more convincing than I'm finding some of the other answers.
    – user541686
    Commented Aug 30, 2015 at 18:16
12

The key issue is not whether or not you remove the outliers, but whether or not you describe and explain what you did. There are many valid reasons to remove outliers, but if you do it, you need to say that you did it and say why.

In contrast to some of the other answers here, I don't think it's absolutely necessary to fully explain the outliers before excluding them. But if you don't have an explanation, you need to say that too. Ideally, you would explain the results both with and without the outliers. If the presence or absence of the outliers doesn't affect the overall conclusion, then you can still stick to that conclusion, while mentioning the outliers as a curiosity perhaps worthy of further study.

Of course, how any of these strategies will be regarded by reviewers depends on your field and publication venue. But those reviewers need to be aware of whatever choices you made in your analysis. Discarding outliers is an analytical choice, and making any analytical choice without disclosing it is unscientific.

Improve this answer
8

As someone with less of an academia background and more of a computer science background, my first instinct for a small amount of tests failing is that these particular tests did not execute properly. Basically, your algorithm didn't complete and returned early due to a bug. This bug can either be in your code, or in your dataset, or in both. Either way, a difference of orders of magnitude is not normal. Check the results of those particular runs and see if they're normal. For all we know, those 5 datapoints might actually be the algorithm running correctly, and those 9995 other datapoints are the bugged ones (unlikely, but possible).

As for displaying these outliers, have you considered displaying that graph with a log(10) Y axis? This would reduce the amount wasted space, but still show that there are outliers.

Either way, removing datapoints for formatting reason is falsifying data, just like you would have if you removed them because they didn't prove your point. It can easily kill your career.

Improve this answer
10
  • 5
    @Mehrdad it might be a little extreme, but it definitely wouldn't help your career. You're basically removing datapoints because they don't fit your idea of what the data should look like. That indicates that you think your idea is more important than reality, which is contrary to what science is all about: generating an accurate representation of reality.
    – Nzall
    Commented Aug 30, 2015 at 18:44
  • Usually hidden or distorted data catches up with the perpetrator with unpleasant results. If you want to take the risk, that's up to you. In general people in your field who are your competitors will be a lot less kind than people on this website. They will take delight in proving you wrong if that's what you are. As others have said, you need to be able to explain the outliers or you are not ready for publication.
    – chasly - supports Monica
    Commented Aug 30, 2015 at 20:10
  • @Nzall By my reading of the question, OP wasn't removing data points because they didn't fit his idea of what the data should look like. When you run a program many times with a variety of input values, you may serendipitously have some data which falls straight through all the if-then-else branches and looping-until-done structures like diarrhea. If there's nothing wrong with those input values, and there's nothing inconvenient about those time results, and including these particular graphics would clutter up the paper unnecessarily, then I don't see the problem with not choosing those...
    – aparente001
    Commented Apr 2, 2018 at 1:47
  • ... particular plots to include in the paper. When we hear "outlier" we sometimes jump to conclusions. It can be helpful to examine what exactly is meant by "outlier" before drawing a conclusion.
    – aparente001
    Commented Apr 2, 2018 at 1:48
  • @aparente001 If you have data which falls through the cracks and gives unexpected results, then you should see why it falls through those cracks and why it gives those results. Extreme outliers might point to a mistake in your data processing algorithm, which sometimes might even change the result of that same algorithm applied to other datapoints.
    – Nzall
    Commented Apr 2, 2018 at 12:27
4

To be pragmatic (and agreeing with the overall philosophy of being fully transparent always) just make your charts in the space provided with the outliers excluded and then put a big darn footnote on the chart explaining that 5 points were excluded due to size constraints on page and offer commentary there about why / how these points exist. Just my 2 cents from one long suffering academic to another...

Improve this answer
1
  • as mentioned in other answers, the presence of those outliers can be interesting on itself, unless its a bug. It should be better to try and show everything properly than to hide them, even mentioning.
    – Fábio Dias
    Commented Sep 25, 2015 at 18:38
4

It's always a good idea to change your plots to make your data look clearer. It's never a good idea to change your data to make your plots look clearer.

There are many ways to tell readers about outliers. My favorite is to use a plot with a conspicuously broken axis, which calls attention to the outliers without disrupting the rest of the plot.

It's good that you're worried about confusing your readers. You should think carefully about how to warn your readers about details like this without distracting them from the story you're trying to tell. However, as many others have advised you, you should never silently ignore data to make your story sound simpler than it really is. Doing this might make your paper a tiny bit easier to read, but only at the potential cost of making your work impossible to reproduce (as Peter said) or throwing away an odd detail that would have turned out to be an important clue (as keshlam said).

Improve this answer
2

If you understand the reason for the outliers, and they represent a defect in your measurement methodology, then correct the error, and do the experiment again with improved instrumentation. You don't need to publish details of all the blind alleys you went down. If you don't understand the reason for the outliers, then ignoring them is unethical, though I'm sure it is very frequently done. If you do understand the reason, e.g. sometimes the algorithm just gets lucky, then you need to mention them, but they don't need to appear on the same chart as all the other measurements if that would make the chart unreadable.

Improve this answer
1
  • Doing an experiment again might cost millions. Doing that for just a few datapoints seems ridiculous and it's better to use statistical analysis then.
    – paul23
    Commented Sep 2, 2015 at 16:00

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .