Impact factors can be very misleading. Journals in chemistry or material sciences have in general higher IF than physical sciences, simply because the community, amount of academic research topics, industrial research, researchers and engineer is much higher. So more people read such journals, although the truly higher impact research is likely achieved and published regularly in physics journals, as nobel prizes even in chemistry are often given to physicists due to groundbreaking achievements in methodology and theory that advance further progress in chemistry and material sciences. This turns everything a bit topsy-turvy when it comes to judging in which journal you should publish your results.
In my opinion you have to guess in which journal you might get the most citations of your work. For a non-tenured researcher this is in my opinion by far the most important criterion/number (so rather pointing towards a gaining a high h-index). Of course it is nice to have a nature/science paper, in some scientific branches it looks even mandatory to be suggested for tenure. But I heard already in germany about a backlash of the publish or perish culture and that tenure committees again look more on the quality of publications than the sheer number or IF due to citation cartels etc. making quantities like h-index etc. inaccurate to truly judge researchers.
If you have tenure and don't need so much to networking anymore focusing on high IF journals looks common because a publication in nature/science gets more media attention/coverage than in a field-related journal and acquiring funding becomes easier with high IF publications. For non-tenured researchers or results that are really theoretically and experimentally original, personally I would always choose the journal where you guess you will get the most citations. That's in your interest and in that of the community. E.g. gravitational waves discovery could have been easily published in nature/science, but for above reasons physical review letters journal with much lower IF was chosen as this journal gets by far the most attention among physicists.