I have a master's in applied math and want to get a PhD in pure math. I'm applying to universities that don't require GRE, GRE Subject and IELTS/TOEFL for fluent English speakers.
From the context of the rest of your question, it seems clear that you are mostly applying to pure math PhD programs in the US. But in that case: more than half of these programs require or strongly recommend the GRE math subject exam. So I don't understand why you would restrict yourself in this way. It doesn't make a lot of sense to plan your graduate education and future career around not wanting to take a certain exam.
My bachelor's grades are bad and my non-pure math master's grades are bad.
Well that's bad, sorry to say. Having higher grades in pure math in your master's program will help a bit, but grades are much more meaningful and competitive for undergraduate students than for graduate students. (For that matter, what constitutes an unsatisfactory grade is much different for graduate students. In my math graduate program, a student needs to maintain a 3.0 GPA in order to be in good academic standing.)
I've had a lot of research projects in master's (mostly by group)
Group research projects will not be weighted highly by most pure math PhD programs.
and cannot get very strong letters of recommendation from main research project (sort of a thesis, also by group) supervisors.
And that's also bad, sorry to say.
On the one hand, the GRE Subject may make up for low grades or the fact that I haven't had complex analysis, number theory, graph theory and abstract algebra.
Number theory and graph theory are completely optional classes in most American programs. Complex analysis and abstract algebra are not. Not having these courses, especially as a master's student, and wanting to get a PhD in pure math is once again bad, I'm sorry to say. Complex analysis and algebra are courses that most PhD students would take (or place out of) in their first year in the program. If you haven't had undergraduate courses in these, you would be starting behind most students with bachelor's degrees. That you haven't taken these courses as a master's student is yet more concerning: it seems to undermine your stated interest in a pure math PhD program.
The keys to a strong application for pure math PhD programs are:
- Good GRE scores.
- Good grades received in courses which the committee will recognize as either good preparation for graduate courses or the equivalent versions of basic graduate courses.
- Strong recommendation letters.
So I'm sorry to say that your application sounds like it will not be strong on any of these fronts.
How does one boost PhD admission profile...
You boost your PhD admission profile by improving on 1. through 3.
Nothing that is as important as what I've said above.
Is it possible to work for a professor for a research project?
Yes, but in pure math PhD admissions, research projects are not weighted very strongly at all, as has been discussed thoroughly on this site. Especially, the very few students who can do serious pure math research before they begin a PhD program will (with very large probability) be exceptionally strong on the metrics 1. through 3. above anyway.
Or pay a professor to supervise my own research projects?
Wait, what? Definitely don't do that.
But on the other hand, the GRE Subject doesn't seem to show ability for research...
Admission to a PhD program is not a debating contest that can be won if you find the right argument. Overall you seem to be asking how to spin a poor application into a good one. Well, you can't...or certainly, you can't if that's what admissions faculty realize you're trying to do. If you want to convince us that the metrics we're using our faulty, here's how to do it: get a PhD in math, get a job at a university with a math PhD program, join the graduate admissions committee, and argue that applications should be weighted differently or that certain metrics should not be used at all.
Even if I were to study the basics* of complex analysis, abstract algebra, number theory and graph theory I'm not sure that that translates to a much higher GRE score. Does it?
I thought you hadn't even taken the GRE Subject exam, so I'm confused when you talk about a "higher GRE score." Certainly I would advise you to take the GRE Subject Exam now (or ASAP; I think it is too late for the 2017-2017 admissions season) and see how you do. It is certainly possible to get a good score without this coursework. Whether it helps to increase your GRE scores is a kind of irrelevant question to my mind...it helps your application a lot to take these courses anyway, and it helps you to take these courses and know this material (some of it more than others, as mentioned above).
*I mean come on, of course I'm just gonna use Schaum's Outlines or a GRE Subject book instead of actually learning from a textbook,
(Probably not the point, but: although students treat "Schaum's Outlines" as being the mathematical equivalent of Cliff's Notes, many old-timers know that you could do worse than trying to learn a subject from these books. Gian-Carlo Rota famously wrote on this subject, praising several of the books in the series. When I taught linear algebra a few years ago and spoke to one of my older colleagues, he lent me his copy of the corresponding Schaum's Outline: it was very nicely done in many respects.)
Why "of course"?!? At this point you've halfway convinced me that you don't actually want to be a pure math PhD student. You seem to have one or more unstated assumptions that are strongly undermining your stated assumptions. I can't tell exactly what they are, but I suggest some introspection on this point on your part.