I will be graduating this Fall with my undergrad (GPA: 3.66) and am planning to go to grad school in math. Given my early history of poor grades (2 Ds, 1 F), I am planning on only applying to the masters programs. Ideally, I would like to get admitted to my current university's masters program (top 15 US school in math). To help boost my application, I have taken some first-year grad math courses. Additionally, I have been doing research under fairly famous applied math professor (h-index above 50. Though I have heard that h-index is not an objective factor). While I have read that in some grad courses they hand out all A's, is there a way to prove to the admission's committee that I actually deserved the A's? Lastly, what do masters admission's committees look for in applicants?
(US centric answer) Any faculty member at your current university can probably tell you the likelihood of being admitted to their MS program. I'll just guess that it is pretty high.
But I think you may be making a mistake to apply to only MS programs, unless you intend to apply to institutions that don't grant doctorates.
For a doctoral institution, I'll suggest that the competition for a slot isn't terribly different for those intending to get only an MS and those seeking a PhD. The reason is that in the early part of the program the students take the same courses, putting the same load on available resources. It is only later that this changes. (A tradeoff, perhaps, is that while MS students often pay tuition, doctoral students often serve as TAs, providing a needed service to the university.)
My suggestion would be to cast a broader net and don't assume that you would automatically be rejected from a doctoral program. Acceptance would depend on many things, including the view the current faculty has of you and what they would say in a letter.
The old, terrible, grades may have less impact on your future than you think, provided that your trajectory is ever upward and you have demonstrated in later courses that you didn't actually fail to learn essential things from the early ones. Not everything is weighted equally, and potential for success is rated very highly.
Cast a wider net. You may catch something nice.
Masters programs would be easier to get into, since they often don’t burden the department with a graduate stipend, that means you’ll be responsible for tuition most likely.
I would apply to PhD programs if I were in your position. If you performed well on your upper division courses and have a gpa over 3.5, then you’d have a good shot of getting in. In this case, you are more likely to get a stipend and a tuition waver. That means you’d get paid to go to school and if you feel that the PhD is too much, you can also leave with a masters degree.
Edit: didn’t have time for a fuller answer earlier.
Don’t worry about the “free grade” in the graduate courses you took. Truly, all that is important there is that you have demonstrated experience and success in a graduate course, and indeed, at an apparently good Institution. No one is going to look at your grade and say “whatever, it’s just a grad course.”
Most grad schools look to your upper division courses as a measure of how you’ll perform in graduate school. I did terrible in my first couple of years in undergrad; I also failed a class or two, but I did well enough in the latter half to end with a 3.5. I was admitted into a good graduate school (in 2008).
As to your peofessor’s h index, that’s actually rather good. It means that he has at least 50 publications with 50 citations. While there is no objective measure of the quality of research, this means that he is well known in his community, his work is frequently cited, and his recommendation will carry with it some weight.
Grad schools look at several things. You need a good GRE score to get past general admissions, but no one cares past that. Your GPA matters, and yours seems good enough (better than mine was). After that they want to know how you did in upper division courses like Analysis, Algebra, etc, since you’ll be taking more intense versions of these classes, and they want you prepared.
Also having experience with graduate courses is a plus, I took a grad number theory course that helped me, and almost as importantly as the rest, they look at your recommendations. You will be part of the school’s community for the next five or more years as a PhD student, and they want to make sure that you can work with someone on research topics.