13

I am currently working on a research project with my PhD supervisor. The focus of this project is a sub-field that my supervisor has not previously published in and one that I am not familiar with. This sub-field has also been studied in the literature for more than 30 years. From past research experience, I am used to performing a literature review before a research project is started to answer the following questions:

  1. What has already been done?
  2. What is the current state-of-the-art (SOTA) algorithm?
  3. What improvements can be done to the SOTA?

Answering questions 1 and 2 simply involves gathering information from different papers. Answering question 3, however, is a bit more complicated, since this would involve understanding how the SOTA algorithm precisely works, and then identifying an area of improvement.

Moreover, the SOTA algorithm in this case is the result of incremental improvements of a basic algorithm that was proposed 20-30 years ago. Since I am not familiar with the field, then understanding how the SOTA algorithm works would involve understanding how the basic algorithm works and then understanding the incremental improvements that were implemented.

The problem is that understanding and re-implementing the basic algorithm would take approximately 2-3 months. This is because it requires technical background that I do not yet have. My PhD supervisor is aware of this, and because of this, has proposed that we do not bother with re-implementing the basic algorithm or the SOTA algorithm and instead try to approach the problem from a completely different perspective, that has not been previously proposed in the literature, using the technical knowledge that we are familiar with.

While I do appreciate my supervisor's ambition, I have a few concerns:

  1. This field has been studied for over 30 years. Complete books have been written about it. The fact that nobody bothered to go down the path that we are exploring strongly suggests that it is not a fruitful one.
  2. Because we would essentially be ignoring the literature, we would be re-inventing the wheel a lot of the time. This means we would be making a lot of mistakes, learning from them, and then most likely end up implementing what was already done in the literature. This would waste a lot of time.
  3. Even if we do end up designing an algorithm, there is no guarantee that it will improve on the SOTA. If it doesn't, then what?

Are my concerns justified? For my third concern, I am not sure if improving on the SOTA is always the goal of publication, and it is possible that some conferences will appreciate a new approach to an existing problem, even if it doesn't improve on the SOTA. However, I am not sure about this.

14
  • 13
    Taking 2-3 months to understand a field new to you seems to me to be quite a minor investment, and perhaps the deeper you go the more you'd find 2-3 months is not sufficient to follow 30 years of work. However, I don't think anyone here will be able to tell you whether it's a better use of your time to start off on a different direction; if they knew, they'd have published something in that direction!
    – Bryan Krause
    Aug 25 at 14:52
  • 17
    As a mathematician, most of the projects I spend time on fail and do not result in publication. Aug 25 at 15:24
  • 14
    A novel attack on a problem can be interesting and valuable even if it is not (immediately) competitive for real-world applications - the different perspective may provide insights or open up new avenues. You don't need to have a comprehensive understanding of equine physiology to invent a car.
    – avid
    Aug 25 at 16:43
  • 9
    "The fact that nobody bothered to go down the path that we are exploring strongly suggests that it is not a fruitful one." Or they didn't think of it, or they had other things to do. 30 years may not be a long time, depending on how active the sub-field is.
    – kaya3
    Aug 25 at 18:14
  • 4
    If it is a mature area, probably you can ask an expert why nobody has tried a given direction. I am sure there are many forums you can ask. As many others have mentioned, there is always an opportunity for an outsider to come in with a new tool and fresh ideas. However, it is also possible that it is a dead end that nobody bothers to tell you about. Personally, in my brainstorming phase, I ignore the literature. Then I check the literature, and adjust my ideas accordingly.
    – VitaminE
    Aug 25 at 21:11
36

Let's start with your concerns:

Number 1: "The fact that nobody bothered to go down the path that we are exploring strongly suggests that it is not a fruitful one." - No. There may be many reasons why this has not been tried, e.g. fashion, lack of relevant expertise, or simply the tools for this route were not available at the time the topic began to develop and people pursued the - then - easier route.

Number 2: Ignoring the literature is not really an option on the long run, but it can be good idea to try making your own mistakes first before being too much "confused" by previous approaches. Once you tried and failed a few times, you will be able to appreciate much more the literature and understand much more easily why they did things the way they did - or, if you succeed, you now can try to understand where they failed or how they did things differently. It can be in general much easier to read the literature if you had the possibility to try a direction on your own. Ultimately, you will have to embed your insights into existing knowledge, but if your method is superior, it will supersede the work, and if it is not, you still may have understood something beyond the other approaches.

Number 3: Scientifically, it is perfectly ok to try a different algorithm principle, even if it does not supersede high-strung algorithms developed over decades of refining and improvement. The challenge will be getting it published, because some venues believe only in benchmark-breaking methods rather than novel insights and understanding. However, if you believe in the direction, go for it. You might want to try and get a sample implementation of the standard method, though, to be able to run comparisons with the existing models. I strongly recommend getting one, even if only as executable if nothing else.

4
  • 13
    Hmm - publication culture in mathematics is different. Something that's an incremental improvement to a known method using known techniques could only get published in a mediocre journal, whereas a new idea for attacking a problem that has some success would be more highly valued even if it doesn't achieve as much. Aug 25 at 17:12
  • 11
    @AlexanderWoo Yeah, I know. Unfortunately, Computer Science has in some fields now moved to "performance" criteria and if they are not satisfactory, that supersedes originality. Aug 25 at 17:14
  • 12
    I would also add that the criteria for a good publication are slightly different from those of a good PhD thesis. If you show you can perfom original new research, this makes a decent PhD thesis, even if this research does not improve on the state-of-the-art and has no direct application.
    – quarague
    Aug 26 at 7:38
  • 2
    @quarague I think novel ways of thinking should be publishable. Olympiad of performance is ok, but science is more like ice skating, there is a grade for the nonmeasurable aspects of the performance. Also in a publication, I would ask where's the insight? If it's just about improving an algorithm with some obscure trickery, you can as well work for a company. Aug 26 at 11:22
18

I think your title is a bit misleading: "not wanting to invest 2-3 months into reimplementing an algorithm" and "not caring about the current literature" is not the same.

To the point. First, I don't see why an algorithm X must be necessarily constructed as an extension of an algorithm Y. You have to care about improving the results provided by SOTA, but it doesn't mean building on top of a SOTA algorithm. Second, you have a concern that your method won't be better than SOTA. It is of course valid, but you may as well fail to improve an existing solution. Third, I find it odd that in a well-research field you need to "reimplement" anything. Can't you get some ready-made implementation and work with it?

In general, I am afraid there is no "guaranteed" road in your case. There is some risk in both options. In such situations I usually try to find at least some "sellable" features of my solution, which can be considered along with SOTA results. For example, you can consider whether your approach is going to be faster / use less memory / less CPU / be better in certain specific cases, etc. This should give some backup options.

4
  • "but you may as well fail to improve an existing solution." - What do you mean here? Also, "I find it odd that in a well-research field you need to "reimplement" anything. Can't you get some ready-made implementation and work with it?" - If I want to improve on an existing algorithm, then I should understand it very well. One way of doing this is to implement it myself.
    – mhdadk
    Aug 25 at 22:51
  • 2
    1) I mean that you might fail to improve the existing algorithm. Try to improve quicksort, for example. 2) You are talking about an algorithm that would take months to implement. You won't really improve all its pieces -- I bet you'll focus just on some of them. So you'll need to understand only these pieces well. "One way of doing it is to implement" -- right, but we can't all write our operating systems or programming languages to understand how they work, I think you are really aiming at doing it the hard way. Aug 26 at 14:49
  • 1
    @mhdadk A recurring topic here is a paper comparing old method A versus new method B, where peer review finds that the comparison is invalid because the author of the paper implemented old method A incorrectly. (Of course it's also possible to find an existing implementation of old method A which is incorrect, but maybe if it's been around that long then there is one which is already well tested--or at least it will be less awkward for you if it's some other well known thing that turns out to be wrong vs your hand written thing.) Aug 26 at 22:50
  • 1
    @user3067860 Even more, even when the old method A is implemented correctly, a reviewer may still believe that more time was spent on optimizing the new method's implementation, making the comparison stale. For this and other reasons, in some CS communities, it is standard to compare different implementations, where the authors of the new approach only implement theirs. If the new implementation of the new approach is better in some way, this is used as a signal that there's some merit to the new idea.
    – DCTLib
    Aug 27 at 9:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.