How to get better at identifying research gaps?

Question

One of the crucial skills of a scientist is to identify research gaps. Unfortunately, students rarely receive much formal training in this skill during their studies. How can one get better at identifying research gaps?

---

Some problems that I have seen which make identifying research gaps difficult:

• The best problems are already identified or solved by other people.
• Authors of publications sometimes avoid being overly critical of their own work, and try to phrase their publication so as to make it seem definitive, and avoid admitting that they failed to explain a phenomenon.
• Some scientists are wary of being scooped, and avoid mentioning what they see as future projects in the discussion section.
• Due to publish or perish culture, there is pressure to split research efforts into minimal chunks - such chunks may seem like they are steps along the way to a solution, but in reality, trying to work on the apparent next step is risky because the author is likely to scoop you.
• High-profile journals such as Nature, Science or PNAS probably prefer papers which answer many questions, rather than presenting many uninvestigated open ends.
• Lesser journals may have publications that have only solved part of a problem, but those may be buried under a sea of uninteresting publications. (also the lesser journals are much more numerous and harder to get abreast of)
• It is easy to confirm that an idea has already been done, but it is difficult to confirm that an idea is truly novel.

Answer 1

Finding good research "gaps" is indeed a challenging and important task. It has different components that all require slightly different skills.

Above all, a useful research gap is interesting, feasible, and unexplored:

• interesting: I think this part goes without saying. But how do you determine 'interestingness' ? This is a complex mix of structural aspects of the field and community opinions, so it varies a lot from area to area. But probably the best way to understand what's interesting is by talking to people, asking them what they think is interesting, and why, and also understanding how even the basic questions in your field came to be deemed interesting (an advisor/senior researcher can often help with the latter).
• feasible: It's not hard to list the "top 10 open questions" in your field. And for some researchers, that constitutes their entire research program. But that's not realistic for most people, and especially for students. Feasibility is about determining whether the tools exist to even make a dent on the problem. This requires a lot of experience in using different methods and understanding what they can and cannot do for you. It also involves thinking somewhat "sideways" about tools and problems, because it's likely that the obvious approach to an interesting problem has been tried. You will have to read papers very closely to find little gaps, or missing reasoning steps, that might suggest that a technique could get used. And you have to learn how to "fail quickly" when trying a new idea.
• unexplored: While it might be educational to work on a new direction that has been explored before, you don't want that to be the fate of all your research explorations ! Here again, being plugged into your community and understanding what people have tried (or not) is also helpful. There are often "folklore" statements of the form "Oh this doesn't work because...". Sometimes it's true. and sometimes it's not. But it's best to understand what people have seen and what they're trying. Related to this is the idea of knowing your strengths: you'll accumulate a set of tools over time that you're an expert at. Make sure that you can use them: that gives you a competitive advantage.

We get a lot of questions here about whether anyone can do research (even if they're not in a formal academic environment). The answer of course is yes. But where amateurs (in the sense of people not being paid to do research) can often slip up is in identifying gaps that are feasible and unexplored (finding interesting gaps is usually not that difficult to do on your own) because they're not plugged into a larger commmunity.

Update: In response to the edit, here's an interview with John Baez that directly addresses the first point (of all the good problems being taken). John Baez's other suggestions are very good as well, and form a nice 'counter' to the Hamming article linked above.

Answer 2

The only way I know is to read, read, and then read some more. The more you know and truly understand about your field and other fields, the better you'll be at identifying gaps. There is no shortcut to this. It takes a lot of hard work and experience.

Answer 3

Gaps could be a hindrance when creating a change in practice because there is no research to support the change or the use of an intervention. A thorough review of literature should be performed to identify validity and reliability before considering implementing a change in practice. A critical part of reviewing literature is identifying gaps, and when they are found an outline should show those areas of research that is incomplete. Sometimes gaps are a help if further research information can be found on the topic. So, there is a potential for further study or independent research. Also, a researcher can produce their own interpretation and suggest how future studies may carry on or what actions could be taken to fill those gaps to improve the topic outlook.