While essentially seconding StasK's points, I'd be inclined to make a stronger claim about the utility of looking at not-so-elementary physics. That is, in addition to all the "mechanics" applications of the 18th and 19th centuries, and Maxwell's late-19th century electromagnetism (which provided a huge impetus to ideas about vectors!), many aspects of quantum theory focused attention on differential equations which have proven to be important examples of mathematical phenomena, in addition to applications to physics. This important-example-intensely-studied phenomenon continued with Bargmann's and Wigner's studies on representation-theory of specific Lie groups, which provided the backdrop for Harish-Chandra's vast program. This specificity was in marked contrast to the "generalism" that mathematicians of the time were embracing, e.g., Weil, Godement.
I think it continues to be the case that physical considerations suggest very-specific examples meriting intense study... which provide test cases for "purely" mathematical ideas.
Witten's (and others') relatively recent "physics" programmes have had a large impact on algebraic geometry (moduli problems, mirror symmetry).
Although I'm also fond of the crypto application/motivations of algorithmic number theory, the breadth and depth seems not as great as the math-phys connection, although of course the elapsed time is much less than for math-phys.
(The optimization and math econ, and comp sci and category theory applications/connections are less familiar to me.)
Still, I must confess that I dropped an undergrad physics minor while studying mathematics, because it seemed dreary to me at the time. Partly this was due to my inability to see the physics ideas underlying the tricks to evaluate integrals, but perhaps partly due to the accidentally-dumbed-down viewpoint promulgated in the physics courses ... presumably aiming at "accessibility".
And, yes, there is a similar common risk/disappointment in mathematics courses that give up ideas for the sake of "tractability". The risk is that it gives the wrong impression.
(Yet, yes, sometimes I've been told that what seems to me insanely fussy detail-mongering is the very essence of mathematics, and that perhaps I insufficiently appreciate "proof".)