I am close to finishing a PhD in applied mathematics and I'm looking to develop future projects that go beyond my dissertation topic into other domains. One of the major subjects my advisor has suggested that I look into is an extremely saturated - but still largely "unresolved" - field. While I find the subject incredibly fascinating and challenging, I am also intimidated by the amount of work that has already been done, and I worry that I might loose the niche I've dug out for my dissertation by trying to play in someone else's big pool. I feel like I might have something to contribute, if nothing else from a "translational/interdisciplinary" perspective (applying the expertise gained during my PhD to a different field), but is there a good way to feel out where exactly I fit in? I suppose a logical answer to my own question would be to identify a post-doc position with a leading group in the field, but I am interested in hearing other opinions.
If you browse through top 10 schools (or researchers') websites, you might get a "feeling" on where (or what) their research is heading towards! For example, you can check what their newest post docs are working on, PhD students' thesis titles etc. You can also do the same thing by checking NSF newly funded proposals. This may not give you a definite answer, but it might help you establish a "directional sense"
PS. I'm in engineering not math.
This is a mathematics specific answer.
I have the vague feeling that what your advisor means by "look into a subject" is different from what you think he or she means.
Most mathematicians have many failed projects in their file cabinets; in fact most probably have at least three and maybe ten failed projects for every successful one.
So, when your advisor suggests looking into this new subject, perhaps all they are suggesting is spending a couple weeks looking at some of the problems in this subject, looking at all the approaches that have been tried, and thinking about whether, from your different perspective, you have any different ideas that have some chance of being successful. If you think you have an idea, you can spend another couple weeks on it to try to see if it helps. If you don't have any ideas, or if your ideas turn out to not go anywhere, then you move on to a different set of questions.
All this costs you is a few weeks of time, and even if you don't have any ideas right away, you learn more about an important subject that you didn't know so well before. Maybe you don't have any ideas for this subject, but maybe you'll learn enough about it so that you can apply some ideas for this subject to a third subject.
Edited to add: In the context of your research career, this means you don't have to leave the niche you've dug out with your dissertation. You can continue working on questions related to your dissertation, and in addition, you can spend some of your time thinking about how what you know might contribute to this other subject. Where you fit in in this new (to you) community will naturally be determined by the actual contributions you manage to make (noting that not all contributions come in the form of a paper).