I am a physics undergrad who has worked with profs mostly on areas in Quantum field theory, string theory. However, my interests have changed slightly over the areas, and now I want to pursue a PhD in Pure Mathematics, perhaps in algebraic geometry or topology. Is it OK, if I apply for a math grad school with recommendation letters from physics profs, or would this diminish my chances to get selected? Should the recommendation letter be given by a prof working in the same area as that you want to apply to?
I agree with blackace that you should probably be OK, but let me sketch what some of the drawbacks are:
Ideally, an application to math grad school will have recommendations from mathematicians. The further you get away from that, the less meaningful the letters are. (For example, at least once per year I see a letter from an English professor, which is utterly unhelpful.) The basic issue is that you need recommenders who really understand what it takes to succeed in math grad school and as a mathematician. Fortunately, physics is close enough that physicists can do a pretty good job of judging this, so you should be OK. In my experience, the admissions committee will worry about two things:
One is that physicists may not appreciate certain math-specific issues. For example, the expected coursework and background. A physicist may not fully understand the extent to which someone's background is nonstandard or deficient for the math program they are applying to.
A second reason is the belief that most people's standards go down a little when making recommendations for other fields. If someone is applying to the top schools in your field, you know very well what the standards and competition are like, and you have something invested in the system and your own reputation as a recommender. In practice, recommenders from other fields seem to be a little more cavalier about making strong recommendations based on a feeling that the applicant is smart, rather than a comparison with the rest of the applicant pool. This means recommendations with be taken with a grain of salt.
So if you have equally good prospects for letters writers from math and physics, you should choose the mathematicians for math applications. On the other hand, a physicist who knows you is still a good choice, much better than a mathematician who doesn't know you. (But a mathematician who doesn't know you is a better choice than an English professor who does.)
No I don't think you need to worry about it as long as you have a good track record and evidence for your interest in the other field which is demonstrable. If you get good recommendation letters (not the boilerplate type) from people who know you well and can provide evidence for their recommendation and can talk about your merits objectively you will not be at a major disadvantage. Of course it would be great to have people who can provide recoms in the same area but this is a common thing for people to change their field when they pursue higher level degrees so you will not be the first one having this issue.
It would definitely not diminish your chances if you get strong recommendation putting you in top percentiles of your program and supporting you in your decision. It also comes down to having a very good statement of purpose and explaining in detail why you are interested in changing field and painting a clear picture for your reasons and why you think you will be capable of doing what you want to do it. I have done this personally twice and encourage you to pursue your interest because in graduate school if you want to be successful you really need to be interested and love what you do.
My advise is to talk with your profs and explain your decision for changing your field. They will most likely support you and provide justification on why you can manage (if they think you have the capacity). Good luck!
Edit: I also want to point out that I have had many friends who have jumped from Math to Physics and vice-versa (and also to CS).