I had taken a course in mathematics previously and scored a GPA of 7 in 10. Presently I would like to take a summer research internship related to that subject and was wondering whether it was wise to try to ask him for a recommendation. The other courses I took were not in the field I want to do internship in (except one other, in which I am already getting a letter, but I need two). The reason for the low grades was I messed up one of the exam due to lack of time, but I still believe it was one of the subjects I knew well enough. I am presently in my undergraduate third year. So what do you think is better, not asking for recommendation because most probably he will reject it or atleast trying to ask?
You might increase your chances if you find another recommender that can say better things about you. Ideally, someone with whom you already did an internship or project, who would better know how you do research.
If you have not worked on any internship/project yet, it might still be better to ask a professor from a course in which you got a better grade. Maybe there are some basic/fundamental courses you took during your first year and whose topic is relevant for the internship.
If you still think that this professor is your best option, the argument "because most probably he will reject it" is not a reason to refrain from asking. Just go ahead and ask: the professor might agree or not, but you have nothing else to lose.