Your question is phrased in rather general terms. There are different kinds of postdoctoral positions in mathematics, and two mathematical postdocs hired side-by-side in the same department may have different career goals. So advice which is guaranteed to be applicable must be rather general. Here goes:
- What is the point of a postdoc in mathematics? I think it is this: the position of "tenure-track assistant professor" at a US math department is really much different from what it used to be. It used to be an extended apprenticeship combined with an audition for a permanent job. Now the job market is so competitive that institutions at all levels tenure the vast majority of the assistant professors they hire: I could easily name hundreds of people who have gotten tenure over the last ten years; I'm not sure I could name ten who were denied tenure. The "apprenticeship" that used to take place at the beginning of these positions is now somewhere between immensely abbreviated and completely absent: in my own department (UGA) we have "graduate faculty" status, which must be conferred separately, but every tenure-track hire in last eight years has been given that status immediately upon arrival. In my department you cannot tell the assistant professors from the associate professors (and from some of the full professors) without your scorecard: the job is really identical, including supervision of PhD students.
Even most brilliantly talented and competent PhDs need time to perform the laundering process from student to faculty. In particular, 99% of the time it just doesn't work for your PhD advisor to have just gotten her own PhD a couple of years ago. This laundering process is what the postdoc is for.
- How does a postdoc work towards the goal of tenure-track readiness and marketability? In what follows I will assume that the desired tenure track job has substantial research expectations (keeping in mind that "substantial research expectations" is still a continuum).
To attain your goal of a tenure-track position, you need to concentrate on two things:
- Put out product -- accepted papers. Both quality and quantity matters.
Many people get a PhD in mathematics before having a single accepted paper (I did), and if they have an eminent advisor who goes to bat for them, having no papers need not be much of a strike against them in the postdoctoral market. Things change tremendously when you apply for tenure-track jobs: you need to show conversion of your research promise into research success. With multiplicity. Publication standards vary significantly among mathematical disciplines (topologists and algebraic geometers publish fewer papers; combinatorialists and applied mathematicians publish more) but I think that at least one publication per year is a reasonable lower bound for almost everyone.
Most math postdocs have not published all the work that appears in their thesis before graduation (my understanding is that this is almost the complete reverse of what happens in most other STEM fields). There are good reasons for this: mathematics is such a technical, specialized field that you spend 90% of your time as a PhD student attaining mastery of one specific subfield, and the PhD is awarded within a very short time period of that attained mastery (in some cases, it is awarded when the advisor feels confident that such mastery lies in the student's near future). So many if not most math postdocs spend the majority of their research time in their first year working on the papers that come from their PhD research. Why does it take so long given that the thesis has already been written? One reason is that they are usually not just transcribing the work but also significantly improving it: again, this goes in hand with the observation that as soon as you know what you're doing you're out the door with a PhD. If you compare a PhD thesis and the paper that it results in, you often see stronger results, simpler and better proofs, and so forth.
Spending time in your postdoc working on publishing your thesis can be tricky: you are in a new environment with new distractions, and after you give a talk or two about your thesis work, many of your research conversations will be about the research interests of those around you, who will probably at least want to involve you in seminars and may in fact want to collaborate with you on further projects. And it's good to do that...but not at the expense of getting your thesis work done. There is even a bit of a trap here, since as a PhD student you get so much care and attention on every aspect of your work, including someone whose job it is at times to tell you to slow down and be more careful: that you haven't yet fully nailed down what you thought you had. Most postdoctoral advisors don't work as closely with their postdocs as thesis advisors do with their students, and when they do it is probably not on the topic of their thesis (they may not even have enough expertise on that to work as closely as the thesis advisor did). So the first priority of any postdoc is to make sure that they actually write up for publication all the work that was "promised" in their thesis, and the second priority is to figure out how to continue this thesis work in a meaningful way.
- Acquire a research program -- distinct from your thesis advisor's program.
I think this is really the hardest part. It is a well-known fact that the most common number of MathSciNet publications for any given author is one. In some cases this just means that some bright young person passed through mathematics on the way to something else -- e.g. this one-hit wonder turned out fine -- but it is distressingly common to see one strong, cutting-edge publication followed by dead silence. What this means is that the work was -- in some suitably high-level sense, at least -- really that of the advisor rather than the student. A lot of postdocs follow the previous bullet point -- i.e., they get their thesis written up -- and then they just can't see their way to a continuing research program. In some cases this means that they still publish some papers, but they have just switched from being one person's apprentice to a different person's apprentice. This is still not good enough for a research-intensive academic job nor, increasingly, a job at a nationally reputable liberal arts college.
It is here that I must disagree with another answer to this question, which says:
In my opinion, you are best looking for a postdoc in an area that is as far removed from your PhD areas as you are willing to tolerate.
That really doesn't sound right to me. It is a relatively easy path to give up on what you've already done and start fresh on something unrelated, for which you can get local supervision. As above, this is not good enough. You want your research program to be considerably deeper at the end of your postdoc than it was when you got your PhD. Sometimes that involves a significant course correction, yes: in my case I spent less than 1/3 of my time as a postdoc working on the subject of my PhD thesis, but the other material that I worked on was material that I had already been interested in and even started writing up for publication as a student. Moreover the two areas were related because I was interested in both of them, and that allowed me to work towards closer relations. Finally, by the time I gave job talks I was able to talk about Hasse principle violations for Shimura curves (my thesis was on Shimura curves) alongside Hasse principle violations for torsors under abelian varieties (which is what I spent most of my postdoc working on). Moving to something "as far removed from your PhD areas as you are willing to tolerate" would have moved me several orders of magnitude farther away, just as I was experiencing a rapid improvement in my understanding of the part of arithmetic geometry in which I did my thesis work. (Eleven years after my thesis, I am now also exploring parts of mathematics that are considerably farther away but I still do some work on things closely related to my thesis.)
A research program has some breadth to it as well as depth. But the ideal way to develop breadth is to expand outwards from your current position, not give up and move to a new position chosen for its substantial distance from your current one.
- It is good to have done more than one thing, or at least to know more than one trick.
I want to reaffirm that after establishing depth, the next thing hirers look for in a research program is indeed breadth. If you're working on what looks to all outsiders like exactly the same thing now as you were three years ago, that's less than exciting, and it creates concerns about your future trajectory: mathematics does move on, and long-term colleagues grow together in their research interests and expertise. However, if you're working in a hot, technical area and doing well with it, then you probably don't want to diversify out of that area as a postdoc, but you do want to take steps to show that the potential for diversity is there. Often digging deeper in exactly the same spot requires learning new tools, even basic things that early career graduate students in other fields learn. I think it's important to continue that kind of learning as a postdoc (and beyond) and show that you're doing it: for instance, in any job talk you should showcase the variety of knowledge and techniques that go into your results, especially if the results themselves look narrow and specialized.
2b) Teaching Teaching is challenging for a postdoc because for a certain range of intended careers it is less important than research...but it is still important, and in fact increasingly so. You have to crank up a different kind of professional vibe to be a successful teacher as a postdoc: whereas for research you are trying to burn the midnight oil to make the transition from excellent to outstanding, what a disaster it would be for you to vow to be the best calculus teacher ever and devote all your time and energy to that. (Part of the disaster is that vowing to be the best calculus teacher ever and putting three times as much time in as your office-mate will not necessarily result in teaching evaluations that are any better than hers: unfortunately they might even be worse.) Rather you need to set a more temperate teaching mark and learn to hit it while leaving plenty of time for your research.
Your specific teaching goals will depend upon what kind of tenure-track job you have. On the other hand, you may be hoping for a job at an elite research institution but may have to settle for a job with higher teaching requirements. A good all-around plan is to showcase at least competent teaching across a fairly broad range of levels, including: calculus courses / service courses for non-majors; courses for undergraduate math majors; and graduate-level courses. If you can cover those bases by teaching fewer courses, that is probably better. In three years of postdoctoral work I taught five semester courses: this was close to ideal, but if I magically had control of my own teaching load, I would have cut it down to four. Because your overall teaching responsibilities are relatively limited, it is good to start thinking about where your teaching letter(s) will come from right away. If you feel that a course is going especially well, by all means invite some faculty in to watch you teach it. That together with good evaluations is sufficient for a strong teaching letter in many cases.
- You've written a lot, but actually everything you've said is fully consistent with "doing as much math [including math teaching] as humanly possible". So I still suspect that doing a tremendous amount of work -- of the right kind -- in a relatively short period is the key to a successful math postdoc. Am I mistaken??
No, you're quite right. Being a math postdoc should not stop you from having a satisfying personal life, but professionally speaking it is one of the most intense, work-intensive points of a mathematical career. There's no secret remedy for that.
One could give a lot more advice for math postdocs, and this has been done. The Young Mathematicians Network has lots of information on this. Google searching reveals that several mathematicians have math-specific postdoctoral advice on their webpages, including Sara Billey and Terry Tao. I tend to think that it is better to talk to people -- lots of people -- and get advice rather than just read about very generalized advice (including this answer, of course). A lot of times you don't even need to explicitly ask for advice: look for young, successful mathematicians (that includes most people who have recently gotten tenure-track jobs nowadays) and talk to them about their experiences. By comparing the backgrounds of multiple role-models you can get a sense of what the "market wants".