For the problem I am trying to solve, I considered 2 base algorithms and devised about 30 variations on each.
I then threw about 2 CPU-years and ran a number of simulations to see how they went.
When that completed I drew alot of graphs and came to some conclusions like:
- Variations of the form
X
do almost exactly as well as the control - Variations of the
Y
family can be predicted to have unpredictable (and thus useless) results - The
d[1,0*]
variation is great under these conditions - The
d[0*]
variation is great under these other conditions
I took the last 2 and made a new algorithm and then tested that and found further useful results.
Now I am going to write my findings up into a paper.
I have 3 kind of results above:
- Points 3,4 (and the subsequent improvements) are interesting and will be the main focus of the paper.
- Points 1, 2, are kind of nonresults. They are failures, they did not produce anything useful.
- For most it isn't even surprising that they didn't.
- For others they are a approach by taken on a similar problem in a paper that inspired me to try and solve this related problem.
So should I comment at all about algorithm variations that were tested and found to not be good?
Why/why not?
Pros I can see:This would help prevent others, not spend time trying them. I have read that it is a problem that in many disciplines (including this one) "negative results are not published".
Cons I can see:Takes up space, may confuse reader as to which algorithm is the focus of the paper.
It feels abit weird that of the 2 CPU-years I spent testing these, and the considerable time I spent making the tests, I will only tell the rest of the world about 5% of my results.