I have wondered about the same question. One thing that I've noticed is that there is an incredible amount of variation in the amount of reviewing done. I have two good friends who are strong, active mathematicians. The last time I spoke to them they were each at least six years past their PhD and had never reviewed a single paper. This seems bad, but it's not their fault: they are not being asked.
I suspect that the lack of coordination among journals in the same field causes the same small number of people to be asked to referee papers over and over again. I am not a journal editor, but I have begun to wonder how editors choose referees: I am someone who is getting (what I think is!) a large number of referee requests. I submit maybe 2-4 papers per year. Not so long ago I used to get that number of referee requests per year, and they were all from people who knew me directly and pertained to topics related to my thesis work or to the areas in which I have written multiple papers. Now I get at least a dozen referee requests a year, and in addition to the above I get some very strange-looking ones. Not so long ago I got asked to referee a paper in (what I think is!) control theory. This was so baffling that I assumed it was some kind of email crossed wires situation and asked the editor about it. It turned out that the paper on real analytic solutions to Bezout equations had been sent to me because of a question on math.SE that I answered, in which I explained why the ring of analytic functions on the complex plane is a Bezout domain. This is distressing: I can see why google would make a connection like this, but in the realm of academic experts, is it not obvious that we are on different sides of mathematics? Experiences like this make me wonder who is refereeing papers.
I'm not sure that counting papers is the right way to gauge whether you are pulling your weight referee-wise. As I mentioned in a comment to another answer, many -- I think most -- papers are refereed by more than one person, either simultaneously for the same journal or sequentially, as many (most?) papers are submitted to more than one journal before they are accepted. Here are some things that I have thought about:
1) It cannot be appropriate for everyone to referee the same number of papers. More experienced people should referee more papers, but also should referee more important / difficult papers. People who have permanent / senior positions in academia (especially: tenure) should have more obligations than those who are struggling to stay in the profession.
2) Not all papers take the same amount of time to referee: not even close, actually. I try to make a point of quickly processing especially short and simple papers: I probably do 3-5 of those each year, and I often process them in a week or even a day. But it would be silly to equate this kind of referee work with the long, difficult papers that you spend weeks or months working on, perhaps having to do considerable outside reading in order to adequately evaluate. Doing 2-3 substantial papers a year is what really keeps me busy as a referee. I suppose I try to keep my eye on the total amount of time I've spent refereeing rather than on the number of papers, although I admit that I don't do this in any formalized way.
3) Unless a paper is clearly of the "one day" variety as above, I try not to let my queue exceed 2 papers. This is also out of respect to the authors: if I have to read two papers before I get to yours, then you'll be waiting for at least six months, probably more. If at any given time every sufficiently experienced/qualified academic had either 0 or 1 papers to referee and made a point of processing papers faster than they submit them, then things would work relatively smoothly, I suppose.