40

I am an MSc graduate and work as an independent researcher. I spent 10 months on a paper. I wrote many scripts in Matlab to implement four other comparative methods as well as a new method that my MSc professor had proposed. Each of these methods was so difficult to implement.

Unfortunately, the method that I was going to propose does not work well compared to the comparative methods, which were introduced already 3-4 years ago.

What should I do now?

  • 60
    This is how I spend most of my time. If you want to do research, you have to be prepared to be disappointed on regular basis. The state-of-the-art usually is what it is for good reasons. Many of your ideas won't work, or won't work as well as you hope. Research isn't like in the movies, where some savant improves an entire field just by having a quick glance over the problem. – Marc Claesen Jun 23 '15 at 7:07
  • 31
    It is of the utmost importance to make a fair comparison to prove state-of-the-art performance. If you cheat, you will get burned. You may get published, you may get cited, but if you get caught (which will happen when someone tries to repeat your experiments) you're in trouble. Integrity is extremely important in research. So in short: NO, that is not okay. – Marc Claesen Jun 23 '15 at 7:11
  • 18
    @Jamaisavenir no, it is not ok to report forged results. – henning Jun 23 '15 at 7:17
  • 16
    @Greg Negative results are still worthwhile results. Chiding people for negative results does nothing worthwhile while encouraging research fraud and p-hacking. If we had more sensible/ethical reviewers and journals willing to publish negative results rather than always going for the "sexy" results then science as a whole would be vastly better off and the OP wouldn't need to treat this like a disaster. – Murphy Jun 23 '15 at 10:54
  • 10
    “I have not failed. I've just found 10,000 ways that won't work.” ― Thomas A. Edison – Scott Seidman Jun 23 '15 at 13:14
46

I recommend that you record your failures and not just your successes. In your case, I suggest you write a technical report giving all the details (the implementation, the comparison, the conclusion). It may not be considered as a publication worthy of your resumé, but at least it serves to document your efforts and may prove to be useful in the future (in case other people are thinking of using your proposed method).

Also note that your method may appear worse when compared to others using a certain metric (say, time complexity) but may appear better when compared to others using a different metric (say, space complexity). You might want to look at your method again.

  • 4
    i agree that a negative result is a result as well. also compare the answers to this thread: academia.stackexchange.com/questions/46300/… – henning Jun 23 '15 at 7:20
  • 1
    In a previous edit OP said this is an algorithm of his research adviser. So, for political reason, he may not want to go around with a big flag stating it doesn't work and he proved it.. – Greg Jun 23 '15 at 7:28
  • @Greg, thanks for pointing that out. To Jamais avenir, if the idea for the method came from your adviser, then you will need his/her permission before you write anything. – Joel Reyes Noche Jun 23 '15 at 8:18
  • 1
    @Greg For other political reasons, the advisor might insist that the student wave a big flag stating that their method doesn't work and he proved it. In the long run, it's always better to admit your failures than to try to hide them. – JeffE Jun 25 '15 at 19:23
  • 1
    @JeffE In principle I agree, and I would recommend that to the SUPERVISOR. However, I don't think the OP should bet on it automatically, or go against the supervisor, if she/he do not share these principles. – Greg Jun 26 '15 at 1:10
33

In the past ten months, you achieved the following:

  1. Validate existing results and implement them in Matlab.
  2. Show the quality of an alternate method.

To me it seems clear that the only way to prevent this time from being a waste is to publish your work or conclusions. First of all, others who work with Matlab may be able to take your implementations and use them for further research. Secondly, if you don’t share your findings on the alternate method, someone else may decide to spend ten months on it before coming to the same conclusion.

That should be sufficient motivation, but of course your negative result can be considered positive in an unexpected way. Either right now by you, or later by someone else.

  • 2
    +1 for "if you don’t share your findings on the alternate method, someone else may decide to spend ten months on it." This is why the research was not a waste of time – even if it didn't find the breakthrough that was initially hoped for. – J.R. Jun 26 '15 at 8:25
14

Next time, perhaps implement only one of the already existing methods and then the newly proposed idea. If the litmus test is that your new idea outperforms all of the baselines, you can test that already early on.

In general, incrementality is key: if you're looking at a task that is going to take you 10 month to realize you need to be aware of the risk you're taking. Be prepared that the results may not end up being what you hope for, or, if you cannot take that risk, find an alternative way of moving forward. Good planning is a valuable skill to have.

For now, would it be possible to analyse in depth why the new method fell short? Is there perhaps a certain sub-problem that it does very well on, in which case you could still extract a (somewhat weaker but) positive result?

Minimally, I'm sure you gained some new insights, and perhaps are even in a position now where you can correct some misconceptions you might have had about your problem: the outcome of your experiment is that something inherent in your problem is not like you thought it was, or else your method would have worked. So you may ask yourself what is it that's different?

If an unexpected obstacle drops on your path, don't just stop. Look for a way around it, or a new direction altogether.

  • +1 for the incrementality suggestion and for the suggestion to analyze why the new method fell short. – Wayne Jun 24 '15 at 15:32
6

The first step is to see if your research is salvageable.

Does your method offer any advantages? Does it work better in certain scenarios? Does it require less/different inputs, and so will be useful where only those inputs are available?

Can you imagine any scenario where it WOULD be better to use your method over the state of the art?

While it can be difficult to publish null findings (i.e. my method wasn't better), it is generally possible to publish findings which include some form of positive results (i.e. my method wasn't better overall, but was better when x).

4

Publishing the results of research allows others to assess these results and build upon them. If the results are negative, it is still worthy for science to publish them because this will prevent others to spend precious time by trying to do the same and failing, without knowing that somebody else has already done it. Negative results may be published without review on repositories such as Arxiv or Zenodo (the later gives a DOI that may improve citability) or, with review, in journals such as PeerJ or F1000 Research (the later even had once a promotion for negative results).

Some may frown upon publishing in relatively low-ranked venues (where negative results would typically find their place), and a reason may be the opportunity cost of spending time to publish the result. I would argue that the time spent for publishing the result is minimal relative to the time spent doing the actual research that led to the negative result. If the time spent for publishing the result would prevent somebody else to spend an order of magnitude more time trying independently the same thing and failing, then publishing would bring an overall positive value to science, and therefore is worth doing.

4

These aren't poor results. Assuming your methods are good, they are good results - they answer a question. They may not be the results you want, but they're still good.

If you don't publish somewhere findable then someone else is going to waste their time unknowingly repeating work you've already done.

Publish

2

Many sub-areas of computer science not only value novel algorithms, but also case studies. While the top conferences and journals tend to favor algorithmic contributions, your can still go for the not-excellect-but-good conferences with a good case study.

So if you can extend your experimental results to a good case study, this is still a good contribution. Often, this mean crafting the benchmarks in a much more careful way than you would do it for showing that your novel technique is good, as you need to convince the reader that the cases that you consider resemble the practice well. So you may need to invest some more time.

1

Well research is all about finding new things and not getting the desired results. Since you planned to achieve something and working on that in the course of your tasks you must have found some undesired results. Its better to focus on what you got and what caused that instead of focusing on what you wanted to get. Hopefully you will reach your goal.

-3

As a researcher all results are Great results, especially when the outcomes are unexpected or under expectations. It is these moments of 'failure' that the greatest discoveries of our times have come from, it is the exception to the 'rule' that have resulted in whole new disciplines and industries. there are no poor results, only poor analysis of the results. Case in point the Michelson-Morley experiment that had completely unexpected results that led to the Special theory of Relativity, and Pauli's discovery of the neutrino because of a mass loss.

  • This might be consoling, but it does not provide an actionable answer to the question. Can you explain how your remarks might help the OP to figure out what to do with their project? – henning Jun 23 '15 at 17:16
  • 1
    All results are great results? Really. What if OP failed to prove proposed method are significantly different than comparative method. A priori we know one method is better than the other. We just don't know which. In that case, OP's tests were not sensitive enough. – emory Jun 24 '15 at 19:18
  • thus proving a result is a great result, in this case the need to find a better differentiating parameter. in other words the test needs to be refined further. or the proposed method is non-sequitor to the problem at issue – SkipBerne Jun 24 '15 at 19:23
  • You've provided evidence that some failures are great results. That's quite far from your claim that all results are great results. It's just not true; 90% of all results are crud (but then, to plagiarize Sturgeon's Law, 90% of everything is crud). – JeffE Jun 25 '15 at 19:19
  • 1
    and this site is proof. – SkipBerne Jun 25 '15 at 19:31

protected by jakebeal Jun 24 '15 at 11:35

Thank you for your interest in this question. Because it has attracted low-quality or spam answers that had to be removed, posting an answer now requires 10 reputation on this site (the association bonus does not count).

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.