Frustration with machine learning and deep learning research

Question

I am doing a M.Sc. in artificial intelligence (AI), and I am collaborating with a computer vision company in a deep learning (DL) research project that will eventually become my master thesis.

Basically, my project is trying an approach that could, hopefully, achieve better results than the approach that is already in use in the company. (For those familiar with deep learning, I try to generate synthetic data for a task where data acquisition is expensive.)

However, after a lot of time and effort spent on this approach I see that the results are terrible. This new approach gives me worse rather than better results than the existing one, which means that at this moment all the effort has been a waste of time.

I feel stuck as I can't make any progress. Every single idea that I have that could potentially lead to improvements makes no difference. This is even worse than if I had worse results as in this case I could at least know what doesn't work. I feel that it is very hard to get any additional knowledge on this task as the experiments that I do don't validate any hypotheses. After months of working on this model, it still seems to me a complete black box.

As a result I end up working much more hours than agreed on in the contract, only to fail in generating any relevant new information or progress that I could report to my boss/advisor. I am spending a lot of time reviewing the code thinking that there must be something wrong.

Sometimes I wonder if my whole project is simply doomed to fail. Maybe this idea is simply not doable for the current task. And even though I feel that my advisor is aware of this danger, I can't help but feel that a failure of this project is a failure of me as a professional. After all, how can I be sure that I tried everything or that I correctly implemented those ideas?

Before this I was sure that being a AI researcher was the career that I wanted to follow. But now I feel that I can't cope with this level of anxiety and frustration. If it is like this for a M.Sc. project, I wonder how it would be for a Ph.D.

Did anyone have a similar experience in DL or even in another area? Can anyone with a Ph.D. share some thoughts whether things will continue to be like that?

Answer 1

As I wrote in this old answer, "it didn't work" is not the end -- the end is knowing why it didn't work.

The caveat with AI/ML is that things are moving so quickly that "nobody got time for" clever ideas that didn't work. In other fields, a negative result might still lead to a strong publication, but this is not the case in AI/ML. Even things that do work but don't outperform the state-of-the-art are unlikely to get much attention. This is the nature of the game, and it is something to accept if you move forward in this field.

After all, how can I be sure that I tried everything or that I correctly implemented those ideas?

This is the main thing I would focus on. A general rule of thumb in research (at least, in fields where something is being created or calculated) is to do many "sanity checks" before running the complicated, conclusive experiment or calculation.

In computer-related research, a mistake that I often see novices making is that they write hundreds or thousands of lines of code and then turn it on (often running it for several days!) and expect it to work. I always suggest the opposite: start with the simplest possible thing (e.g., 16x16 thumbnail images) and verify that that works as expected. Keep simplifying until it does the right thing (and ideally, until it runs very quickly, allowing you to run experiments in near-real time). Then you can slowly add back the complexity. In this way, you can ensure things are implemented correctly, and if it ends up not working, you'll have some intuition for why the original idea didn't work in practice.

Finally, a subject-specific comment: if you are using GANs to make synthetic data, you should know that GANs are highly finicky (e.g., to the hyperparameters). You may need to run many experiments before you find the magic numbers that achieve the desired performance. This will require many GPU-hours -- all the more reason to run many "sanity checks" before unleashing all your experiments.

Answer 2

It's no surprise that many promising ideas don't work out. Indeed

Maybe this idea is simply not doable for the current task.

For a related story in mathematics, see What to do when you spend several months working on an idea that fails in a masters thesis?

"Machine learning" is a trending topic. Lots of smart people are thinking about it, and lots of students are signing up. There will be many deadend trails in the search for the few that really lead somewhere useful. Only you can decide whether you can tolerate the frustration natural in that search.

Answer 3

I agree with everything in cag51's answer. But I wanted to add that your own assessment may in fact be correct; "But now I feel that I can't cope with this level of anxiety and frustration." You are right, a PhD will likely be 4 more years of this. You might not enjoy that very much.

That's ok, you will be more than qualified to begin a myriad of PhD's in neighbouring fields. You are actually at a great stage to be having this realisation. Off the top of my head computational biology often gets people coming in from more computational fields to transfer knowledge. Biology degrees don't always include much background in either computational modelling or big data, so there is real demand for interdisciplinary talent. Also, I suppose it should be mentioned that you could always go into industry for a year, earn a much stronger salary, then come back to this decision. That is not an uncommon route.

If you do decide that on balance this is still the thing for you, then I agree with the advice in cag51's answer. You can make this work, it just might mean a slightly different strategy. And lots and lots of persistence.

On the other hand, I did a masters in physics theory. I did well in the masters, and got good PhD offers in theory, which was tempting. But in truth, doing research in theory was spoiling my enjoyment of physics. So I went and found a computational PhD and loved it. Now is the moment to work out what you want your next 4 years to be, because you are at a very unusual point in life when you are almost guaranteed to get what you chose.

Answer 4

Try these steps:

1. Go back to your basic assumptions (what modifications may yield improvement) and test them out one by one, not together.
2. Start with small modifications of the original state-of-the-art training process and only increase the amount (or amplitude, or sample size) of customizations gradually to see when the results start to deteriorate rather than improve.
3. Remember that when generating fully synthetic data, it may be difficult to obtain the same statistical properties as in real data - this is a common problem of data generation, and it may seriously harm the trained model performance (as measured on real data, of course). That's why it may be better to do augmentation instead: start from a real sample and disturb it in different ways, rather than create samples synthetically from scratch. With this approach, some of your ideas may still apply, and you will have a much bigger chance of actually improving the accuracy, because you can easily fine-tune how much of the perturbation is applied, going gradually from 0% to 100% and observing changes in the performance along the way.
4. Don't train on synthetic data alone, but combine a (larger) sample of synthetic data with a (smaller) sample of real data, to let the model observe and learn the less obvious real-world characteristics, ones that you didn't manage to replicate in a synthetic generation process. Optionally, you can also put a larger training weight on real samples to compensate for a smaller relative size of this subset.
5. Make sure that your evaluation setup is correct and that you're measuring what you really want to measure.

Last but not least, keep in mind that "research" is all about trying many different ideas and seeing most of them fail, just to spot one that actually works. This is particularly relevant for Deep Learning, where the complexity of algorithms is very large and still growing with new advances in the field. Sometimes, the best thing you can do is accept that your idea doesn't work and proceed to the next one. :D

Answer 5

It's all fine and well until

I can't help but feel that a failure of this project is a failure of me as a professional

Indeed, the subtle art of being in academia is digging golden nuggets of knowledge from otherwise "failing" (well, most of the time, anyway) projects. Most of the time, if the spec is detailed enough - e.g. "achieve X with Y precision on Z dataset by time T", it is met by the virtue of the "unknown" portion of the work being already done. That is, no one claims to finish a completely novel and creative work in a given timeframe. New ideas usually don't work and often many man-months are "wasted" on them... That is, until you realize the value there is the knowledge obtained in the process.

The issue is not specific to ML/AI, what changes is you don't have your expectations as high in many other fields; the level of hype around ML has research tied much closely to real world applications where people expect workable results, something that may take decades in more "traditional" fields.

Answer 6

This may not be the answer you (or other people) might want to hear, but I think the real reason is that the performance of an AI mostly depends on primarily the intelligence behind its design and secondly the resources given to it. For deep learning algorithms, this intelligence factor includes the intelligence behind the deep learning algorithm as well as the intelligence behind the heuristics used in encoding the input data in a way that the deep learning algorithms can perform well on that data.

If the previous solutions used by the company had a lot of intelligence behind them, likely due to being designed by intelligent people who imbued some of their intelligence into heuristics on which those solutions are based, they would be able to easily outperform any unintelligent application of deep learning to the problem.

There is no cheat for this; you cannot get artificial intelligence greater than what went into its design (after controlling for resources). So do not be afraid to try to invent your own heuristics based on your own intelligence and your current experience with the problem! Deep learning can be used to fine-tune your heuristics, or help you search for heuristics. Or you may use heuristics to adapt existing deep learning algorithms to your problem. Whatever it is, do not assume that deep learning can somehow create intelligence out from nothing.

The point is, there is something very wrong if every change you make seems to have no effect on performance. It strongly suggests that the deep learning algorithm is so swamped with noise that it is producing some kind of uniformly random results. And this situation is impossible if you were using heuristics, because using your own intelligence you can work through examples of bad performance and tweak the heuristics to make improvements!

Answer 7

In this field not many improvements are achieved with large changes. Often times, the current state of the art is very good. But that does not mean there is no room for improvement. Sometimes, these systems are not well optimized. Or in some cases there are a group of special cases that are more prone to failure. You may attack the problem from these angles.

For instance, if you believe the system is not well optimized, you may try to find more optimal parameters. Start with the current system and slowly add/remove more features, adjust parameters of the current system. Change one aspect at a time to see if any of these make any difference. The difference does not need to be large; smaller improvements can lead to more smaller improvements and at the end of the day you might end up with a significantly better optimized system.

For the second case, you must make carefully controlled experiments to find cases where failure is very common. Try to identify why they fail and integrate alternative detection methods to these cases. You might segment the feature space and if a sample falls into that space, you can use a different set of feature extraction or a classifier system to identify those samples.

Also do not constrain yourself to deep learning approaches only. Machine learning is a vast field with many methods. Try other classifiers or combine deep learning with classical techniques. Analyzing the distribution of the data can open up interesting insights. Do not shy away from formulating data distribution to solve for optimum classifier for the case. DL approaches are quite easy to set up to work well, but often times they fall behind to carefully curated classification systems. Many practitioners in this field think that it is a silver bullet to all problems, but that is not the case.

Answer 8

The idea of a Masters degree in AI is a pretty large reach for a university to award. Most AI is pretty piecemeal and working on magic.

The truth is, despite marketing hype like Hadoop and such, there is no general solution to unstructured data except a full-blown AI solution to correlate data with experiences in the world. Visual processing gives you structure along three dimensions (two spatial). But, much like Edwin Abbott Abbott's Flatland, missing a dimension can force very large errors out of your decision-making process.

You can try to do your learning with more than these dimensions (adding another camera at an orthogonal angle, for example) -- that's one answer. Or, you can implement a full-blown AI model which is ready to be developed in the world and uses a multi-layered, Markov model, for tracking all probabilities on all inputs to create excellent guesses, assuming your input data is ordered (and not noise).

Answer 9

Sorry to be blunt, but I am afraid your question has to nothing to do with machine learning or deep learning - or even computers at all for that matter.

Once you see that what is happening to you is what happens to anybody designing/performing/interpreting experiments, you'll be halfway through figuring out what is not working as expected, and how to make it work. Most importantly, you will understand if this is what you want for your career. In fact:

Before this I was sure that being a AI researcher was the career that I wanted to follow. But now I feel that I can't cope with this level of anxiety and frustration. If it is like this for a MSc project, I wonder how it would be for a PhD.

Doing research is about hitting your head 90% of the time on things that do not work, and patting yourself the 10% of the time it actually works. "Hitting your head" typically means everything from questioning the underlying hypothesis being tested, to the assumptions of the test, to the experiment design, the data taking etc.etc.etc. The ability you need to grow is the methodology to systematically attack all those elemnts separately, and temper the frustration when you don't see the light at the end of the tunnel yet.