In this field not many improvements are done with large changes. Often times, the current state of the art is very good. But that does not mean there is no place for improvement. Sometimes, these systems are not well optimized. Or in some cases there are a group of special cases that are more prone to failure. You may attack the problem from these angles.

For instance, if you believe the system is not well optimized, you may try to find optimal parameters. Start with the current system and slowly add/remove more features, adjust parameters of the current system. Change one aspect at a time to see if any of these make any difference. The difference does not need to be large, smaller improvements can lead to more smaller improvement and at the end of the day you might end up with a better optimized system.

For the second case, you must make carefully controlled experiments to find cases where failure is very common. Try to identify why they fail and integrate alternative detection methods to these cases. You might segment the feature space and if a sample falls into that space, you can use a different set of feature extraction or a classifier system to identify those samples.

Also do not constraint yourself to deep learning approaches only. Machine learning is a vast field with many methods. Try other classifiers or combine deep learning with classical techniques. Analyzing the distribution of the data can open up interesting insights. Do not shy away from formulating data distribution to solve for optimum classifier for the case. DL approaches are quite easy to setup to work good, but often times they fall behind to carefully curated classification systems. Many practitioners in this field think that it is a silver bullet to all problems, but that is not the case.

In this field not many improvements are achieved with large changes. Often times, the current state of the art is very good. But that does not mean there is no room for improvement. Sometimes, these systems are not well optimized. Or in some cases there are a group of special cases that are more prone to failure. You may attack the problem from these angles.

For instance, if you believe the system is not well optimized, you may try to find more optimal parameters. Start with the current system and slowly add/remove more features, adjust parameters of the current system. Change one aspect at a time to see if any of these make any difference. The difference does not need to be large; smaller improvements can lead to more smaller improvements and at the end of the day you might end up with a significantly better optimized system.

For the second case, you must make carefully controlled experiments to find cases where failure is very common. Try to identify why they fail and integrate alternative detection methods to these cases. You might segment the feature space and if a sample falls into that space, you can use a different set of feature extraction or a classifier system to identify those samples.

Also do not constrain yourself to deep learning approaches only. Machine learning is a vast field with many methods. Try other classifiers or combine deep learning with classical techniques. Analyzing the distribution of the data can open up interesting insights. Do not shy away from formulating data distribution to solve for optimum classifier for the case. DL approaches are quite easy to set up to work well, but often times they fall behind to carefully curated classification systems. Many practitioners in this field think that it is a silver bullet to all problems, but that is not the case.