The answer to your question is no, it won't hurt.
How you prepare your application, however, will likely depend on the type of research you want to do while working on your PhD. For your case, if you want to pursue theoretical research, then I think recommendation letters from mathematicians will be a good asset.
If, on the other hand, you want to work in experimental physics, then I still don't think it would hurt, but I would work extra hard at highlighting relevant skills and experiences, such as experience working in a machine shop or with electronics, in other parts of your application.
You still want your references to be good judges of academic performance, however. By academic, I mean that they should work in academia or be experts in research, critical thinking, and other important academic traits in your field. A life-long high school teacher, for example, may not be a good choice.
To summarize, having letters of recommendation from academics outside of your major field should not hurt your chances of getting into graduate school. The letters should speak to your character, work ethic, and natural abilities, not to the skills you possess. Those can be highlighted in your application.
Qualifications: I'm a senior (sixth-year) graduate student in a physics-related field and frequently assist my advisor in taking on new graduate students. We've taken on people with engineering, physics, materials science, and mathematics backgrounds.