41

I've written a classification algorithm that does a pretty good job at classifying some datasets. However, I compared my algorithm to other classification methods, and their results exceed my results by a little. For instance, when classifying a dataset from a repository, my algorithm is getting 95% correct while another algorithm usually gets 99% correct.

Should I continue to publish my results although 1) my algorithm is a little slower, and 2) my algorithm's results are not as good as the other results.

I'm a little torn. I'm excited as my paper and results are a contribution to the classification field as the algorithm is novel. Also, I'm of the stance that you can't beat EVERY algorithm. If we only published algorithms that could (loosely) beat other algorithms either A.) we'd never have new innovations, or B.) eventually every dataset would be 100% classified each time, or C.) every algorithm could instantaneously classify a dataset (speed).

I hope that my algorithm will continue to grow and others will pick it up and extend it. I hope that one day -- with tweaks -- my algorithm can reach 99% too.

I'm afraid of being rejected by the journal again. Yes, my first submission was rejected. One of the reasons for the rejection was that my dataset was small. However, when the dataset was small I was beating the other algorithms. Now, as the dataset has grown, the other algorithms are now beating me. I'd like not to be rejected again.

Improve this question
12
  • 53
    I don't mean to be cruel, but if the state of the art is 99% correct and your algorithm is 95% correct, a reader's reaction won't be that your algorithm is correct 96% as often; it will be that your algorithm makes 5x as many errors.
    – hobbs
    Mar 25, 2016 at 4:15
  • 4
    This is, statistically, the correct reaction, too.
    – Ryan Reich
    Mar 25, 2016 at 4:52
  • 8
    It's simply not true that we'd never have innovations if we only published things that are improvements on the state of the art. You need to give people a reason to care about your algorithm. "If people cared about it, they'd make it as good as other algorithms" isn't a reason for people to care about it: it's a hope about the consequences that will follow if people do care.
    – David Richerby
    Mar 25, 2016 at 6:02
  • 4
    I'd say the paper is interesting nonetheless, because it contributes a new insight on the performance of algorithms. Also, if you don't publish it, the next person who has the same idea will need to investigate it fully, probably with similar results.
    – Simon Richter
    Mar 25, 2016 at 14:47
  • 5
    Just because your algorithm has inferior performance, doesn't mean its discovery is useless. The fact of its existence may be insightful to theorists, and others may see improvements upon yours that propel it to much better performance. You should try to emphasize any aspects it may have besides speed and correctness that are somehow interesting.
    – Superbest
    Mar 26, 2016 at 3:27

8 Answers 8

Reset to default
89

If you want to get technical, in general no learning algorithm performs any better than any other.

The question then, is what can be learned from your algorithm. In your question, you speak of it as a fond intellectual child that you wish to grow and nurture, and you speak of your personal concerns about acceptance and rejection. Here is the thing, though: none of that matters for a publication. What matters is this: what new knowledge or capability is brought into the world with your work on your algorithm, and how can this be objectively evaluated?

Here are some possibilities that I can see:

  • Your algorithm may perform better on an interesting and useful class of problems, and thus be of practical interest.
  • Your algorithm may perform worse, but have some other desirable property, such as executing very quickly or using very little memory, in which case the performance comparison just needs to show that it is sufficient, and you can show much better performance with regards to those other properties.
  • Your algorithm may perform worse, but do so in a way that is enlightening, e.g., taking a more human-like approach, or showing how something can be accomplished by a very unusual and unexpected route. In this case, the performance comparison is simply showing that your algorithm does not perform too badly to be interesting, and the narrative should focus on the path taken to achieve your results and why that is interesting.
  • Your algorithm may only have taught you personally some interesting things about classification and scientific research, in which case you should mourn the passing of a fine research idea and move on with your life.

Only you and those who know your work well will be able to tell which category it truly fits into.

Improve this answer
11
  • 4
    +1 for a good categorisation of what useful results could be.
    – Captain Emacs
    Mar 25, 2016 at 0:58
  • 25
    I'd add another category. You think your algorithm might provide a foundation for other algorithms (it introduces a novel concept). Even if it doesn't pass the first three tests, I'd say at a minumum put it up on ArXiv, who knows what others may find it good for (it might be the foundation for an algorithm in a totally unrelated field). Let others see it, if it is indeed novel.
    – WetlabStudent
    Mar 25, 2016 at 3:39
  • 4
    @WetLabStudent That is really covered by point 3, is it not? "it introduces a novel concept" ~ "showing how something can be accomplished by a very unusual and unexpected route"
    – Angew is no longer proud of SO
    Mar 25, 2016 at 8:17
  • 12
    To add my five cents. I've been to a conference where someone presented an alghoritm which didin't really get better results than state of the art ones but it was so robust, concise and innovative that I was marveled at the pure idea and I could implement it from scratch there and then whereas I can't implement those better alghoritms without any reference which of course takes time to read and process. Take sorting for example I know complexity of sorting alghoritms but then again I usually go with bubble sort because I can implement it fast and go on. The results are not always the key.
    – Sok Pomaranczowy
    Mar 25, 2016 at 9:25
  • 3
    Another important category: The resulting model is more straightforward to interpret than models from competing algorithms (many learning algorithms are essentially black-box models), or that the model uses some domain-specific information/assumption which could make it more relevant to that domain.
    – Bitwise
    Mar 25, 2016 at 12:00
6

Simon Richter wrote in a comment:

I'd say the paper is interesting nonetheless, because it contributes a new insight on the performance of algorithms. Also, if you don't publish it, the next person who has the same idea will need to investigate it fully, probably with similar results.

If "this idea doesn't work as well as one might hope" is the main conclusion then I also see value in publishing it, but you need to consider carefully the venue. Is there something like a Journal of Negative Results in your field?

Improve this answer
1
  • It seems like the author isn't convinced that his algorithm is a dead-end and thinks that future work on it might produce good results.
    – Christian
    Mar 28, 2016 at 18:17
4

Let's take a look at a very common and studied problem: sorting. There are lots of algorithms, starting from very inefficient ones, such as bubble sort up to more efficient ones such as quick sort or merge sort. Of course, in practice, I would like to use the most efficient one, but there are some reasons for which I might choose another one. For example, merge sort might be more appropriate for a machine with sequential access memory. Also, even if I would never use it in practice, there was a point in my life when I studied bubble sort, since it created a softer learning curve for me. Also, I studied merge sort initially not for the actual problem that it solves, but for the method it uses.

Bottom line is, there are many reasons why an algorithm is interesting for somebody, even if it has a lower average performance in practice. Moreover, somebody might find some application for your algorithm where it is better than others (for example, bubble sort is quicker than quicksort on sorted lists).

Improve this answer
1
3

Accuracy is not the only measure for an algorithm: yours may work blazingly fast, be implementable on a microcontroller, classify in real-time or online, serve as a good preprocessing to other algorithms, be robust to noise, who knows. Perhaps it is just very elegant. With a little wit, you can find a way to assert its efficiency.

Many of nowadays standard algorithms (for instance in sound/image compression) contain parts that where not quite state-of-art when published first. But work great in conjunction.

Improve this answer
2

I'll have a go at this with a softer perspective. You write:

I'm afraid of being rejected by the journal again.

What is the worst consequence of being rejected?

Most of us get rejected from time to time, and you are not alone in being worried about that: How do I overcome fear of rejection when writing academic papers?

You obviously have a logical mind. My guess is that if you put that good head of yours into thinking about the Worst consequence, you will find that it isn't that bad. You might even learn something useful along the way. Life will go on.

If not not in the sphere of academia, several famous writers have been rejected: Agatha Christie, J.K. Rowling, C.S. Lewis to mention a few.

Close your eyes and press send! (and if you get rejected: blame it on someone else, that helps ;)

Improve this answer
1

A lot of good points supporting publication have already been listed but one seems to have been missed, even if it might not be relevant for your algorithm: that it is a different algorithm.

I'm working in the numerical department from time to time (for fun, not profit) and just now I'm implementing a function to compute a big-integer nth-root. Although the algorithms I use had been proven to be correct sveral hundred years ago any implementation will have errors: tests are mandatory.

I could have used a second library with a well tested implementation but to avoid errors in the translating of the data types I just implemented two different algorithms: a recurrence (Halley) and a linear function (binary search). The chance that both err in the same way and outcome is very low because the two algorithms are sufficiently different[1].

If your algorithm is also sufficiently different it, too can be used to verify the results of other algorithms. That is not the worst thing, to say the least.

TL;DR: publish!

[1] NB: you may use both for large integers because the binary search algorithm is not very fast for small indices and the Halley recurrence is quite slow for large indices starting at about 300 bits radicand size with my implementation. So your algorithm might even be useful in production, although that point has already been made, if I remember correctly.

Improve this answer
1

Research is not always writing about how your experimentation works perfectly. It's about contributing to the global body of knowledge. Such knowledge can also include what was attempted and why it did not work.

Improve this answer
0

It is possible that your algorithm isn't quite as good as other published algorithms right now, but it may have easy improvements that other people could find that makes it better than other algorithms.

If you publish it, someone might figure out such improvements. If you don't publish it, they won't.

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .