For the problem I am trying to solve, I considered 2 base algorithms and devised about 30 variations on each.

I then threw about 2 CPU-years and ran a number of simulations to see how they went.

When that completed I drew alot of graphs and came to some conclusions like:

  1. Variations of the form X do almost exactly as well as the control
  2. Variations of the Y family can be predicted to have unpredictable (and thus useless) results
  3. The d[1,0*] variation is great under these conditions
  4. The d[0*] variation is great under these other conditions

I took the last 2 and made a new algorithm and then tested that and found further useful results.

Now I am going to write my findings up into a paper.

I have 3 kind of results above:

  • Points 3,4 (and the subsequent improvements) are interesting and will be the main focus of the paper.
  • Points 1, 2, are kind of nonresults. They are failures, they did not produce anything useful.
    • For most it isn't even surprising that they didn't.
    • For others they are a approach by taken on a similar problem in a paper that inspired me to try and solve this related problem.

So should I comment at all about algorithm variations that were tested and found to not be good?

Why/why not?

Pros I can see:

This would help prevent others, not spend time trying them. I have read that it is a problem that in many disciplines (including this one) "negative results are not published".

Cons I can see:

Takes up space, may confuse reader as to which algorithm is the focus of the paper.

It feels abit weird that of the 2 CPU-years I spent testing these, and the considerable time I spent making the tests, I will only tell the rest of the world about 5% of my results.

2 Answers 2


You could omit the uninteresting 95% in the peer-review journal version, but include them as an appendix in the arxiv.org version of the paper. That way they're available but you're not forcing anyone to read them...

  • 6
    Or you could include them in the appendix in any version of the paper, if the journal does not have a space restriction.
    – Wrzlprmft
    Sep 7, 2014 at 8:21
  • 2
    @Wrzlprmft: That sounds good in theory. In practice, all the journals I know evaluate the suitability of a paper in terms of (among other things) the total intellectual content divided by the number of pages. Recently I got a paper rejected from a prestigious journal, and the referee pointed at about 20% of it as being "relatively straightforward" and used that as a strike against it. A journal which really doesn't care if you make your paper 20 times as long and filled with less interesting results does not sound like a good journal to publish in. Sep 8, 2014 at 13:46

Negative results are as interesting, in my opinion, as positive ones. They allow future researches to know what won't work, and prevent them wasting their time (unless they thing you did something wrong- in which case they will try it anyway :-). However, the main paper should probably focus on the positive part, since most researchers have enough to read already and will not care much about the negative results.

In biology at least, it is frequent to mention some negative result and then include "(not shown)". But nowadays with the facilities to add supplementary material there is no reason to not include some proof. I would suggest to include negative results as supplementary material and just mention them in the main manuscript.

  • 2
    Maybe I should have said "negative results can be as interesting...", so we don't dump any crazy experiment we did into the sup. materials.
    – ddiez
    Sep 8, 2014 at 11:20

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .