I'm about to finish my undergraduates studies; I majored in mathematics and minored in physics, and I always intended on going to grad school to pursue a PhD in mathematics, but I've been having doubts recently. I did well in all my courses (3.92 GPA), but I'm trying to seriously consider if my background is strong enough now and if I'd truly have the motivation to stick it out. I've also been thinking even if I decided to give it a shot, it might be nice to take some time off for rest and to improve on some of my weaker areas. But I've been told by a few people that if you want to do a PhD in mathematics, you have to go pretty much right after undergrad, mainly because recent letters of recommendation are so important, and professors forget you after a time. So I wanted to know if this is true, and also thought I'd ask for advice if anyone has been in a similar situation.
The other answers don't really address the issue of recommendations, so let me, at least briefly. I've been on our math PhD admission committee several times and we get many applications from people who've gotten their undergrad some time ago.
First, yes there is some truth to it being easier to get in right after your undergrad degree. The letters of recommendation are important. If your professors know you quite well, and the department is relatively small, they should still be able to write you decent letters after a year or two hiatus, but if it gets to 5-10 years, they may not, and with that kind of time lapse, their letters won't count for as much anyway.
My advice would be to consider what else you want to do. Is there something else you really want to do for awhile (peace corps, travel, interesting job opportunity)? If so, it won't kill your chances for grad school, but you may have to apply to more backup schools. If you're out for longer, it might be best to do a masters first before getting into a PhD program.
If you don't have any definite ideas, why don't you try applying to a few masters programs (Vladhagan's suggestion of trying a masters first is a good idea to give you a sense of what you want to do and give yourself a better background) and a few PhD programs that seem interesting to you? At the same time, maybe go to a career fair and send out a few job applications in the spring? The PhD programs that accept you (at least if you're in the US) at least should give you an opportunity to visit, so even if you're undecided about a PhD in the spring, visiting these schools (and similarly any job interview impressions) may help you make a decision.
But I've been told by a few people that if you want to do a PhD in mathematics, you have to go pretty much right after undergrad, mainly because recent letters of recommendation are so important, and professors forget you after a time.
You do not have to start a PhD program "pretty much right after undergrad". It is most common to do so, but there is a substantial minority of students who are older and/or spent several years out of school. In (American, at least) academia, your age counts for nothing; the ticking clock in the sky keeps track of the number of years since your PhD. I know several people who spent years off from undergrad in the sense of leaving school but clearly kept up with their mathematical reading and learning -- while in the Israeli army, culinary school, creative writing programs... -- and started grad school with skills at least at good as those around them and a maturity that most 22 year-olds lack. I've looked through hundreds verging on thousands of job applicants' CVs, and I am struck by how often the stronger candidates were in their 30's rather than their 20's when they got their PhD.
Of course the biggest risk in taking time off between undergrad and grad is that you will get distracted by the rest of the world and not come back for graduate study. But that's only a risk in the context of your original plan: if you found something else that you like better than being a graduate student, good for you. It is also relatively common that after a fairly small time away -- one or two years -- people realize that they really do prefer an academic career. (For some reason this seems to be most common among high school teachers. Isn't that a bit sad?) If you're not totally committed to a PhD program, taking time off and seeing whether your desire waxes or wanes is a pretty smart idea.
Of all things in your decision, I don't think that going straight to a PhD program because you're worried that your professors will forget about you is a good strategy. Professors don't forget about students that quickly, but after a few years, they may. To combat this, I would say: if you are not sure whether you want to do a PhD right away, why not apply right away to PhD programs? Certainly knowing where you can get in and seeing the programs that admitted you are all factors in your decision. If you apply right away, professors will write letters for you, and if you go away even for a long time, those letters should still be equally usable afterwards: your past undergraduate performance is not a function of time.
At least in my area of the world (Western US) and among the professors from the institution where I did grad work, almost everyone did a master's degree first. There are exceptions to this of course. How I see it is if you are a genius, go on to the PhD directly. Otherwise, a MS can give you some good background without drinking from a fire hose. This is how I improved my weaker areas. I was weak in analysis before grad school. I was able to take 4 analysis classes (graduate level!) for my MS and it was a significant boost.
When I applied to PhD programs, I got my letters from professors I had taken graduate classes from (and my thesis advisor). This allowed them to comment not just on how well they thought I would do in grad school, but how well I actually had done. This also allowed for them to comment on my research. I think it made me a stronger PhD candidate.
I will back up that you will need to be (somewhat) confident that you can become very strong in a specific niche. But that is why you go to grad school; its purpose is to make you strong in your field. And your strengths may change. I entered grad school as a group theorist and left as a probabilist.
I doubled in math and physics at MIT, and went on to get a PhD. in physics. Although I had a successful career, first as a supergravity theorist and then as a computational physicist, I have always wished I had gone into math, which was my stronger love and better talent.
You don't have to be uniformly strong in all areas to do wel in a math PhD program. (Although many nice ideas stem from creative ideas in unrelated parts of math.) You do have to be pretty sure you can become insanely strong in one area, and you do need to be confident that you will love what you are doing.
Entering a PhD program is a major commitment that you should not enter into if you're not completely ready. Your question shows that you're very unsure about what you want to do. Thus I would not recommend entering a PhD program at this point in time.
It is certainly possible to work for a while and then go back to graduate school.
My own personal experience is that when I got my BS degree (in Computer Science), many friends urged me to go on to graduate school immediately. Instead, I went to work as a software developer for the next three years. It became clear that I would need at least an MS degree in order to advance within the company that employed me so I went back to graduate school for an MS in applied mathematics. During my first semester as a full time graduate student I became very interested in a new area (interior point methods for LP), and applied to switch into the PhD program so that I could really immerse myself in that topic. I wouldn't recommend this approach to everyone, but at the same time, I'm quite certain that it helped me to have worked for a while before going back to graduate school.