That is an excellent question and answering it will help me sort out my thoughts on the matter. Which exactly matches the best reason for me to referee papers: because doing so will likely teach me something.
To go into more details, there are basically two sets of reasons to accept or decline to referee a paper.
I assume here that you feel reasonably competent to do the refereeing; otherwise you should simply decline.
1. Reasons centered on the community.
1.a You want your papers refereed, and for the system to work it needs everyone to participate. It is part of our job, and thus we should do it; the big question is where to place the bar.
A good reference point is to consider the number of referee reports one has benefited from, weighted against the average number of coauthors.
This gives good incentives: it means submitting too boldly and facing rejection might "cost" you to force yourself into one or more referee reports (depending on the journal you submit to). Another common reference point, which is quicker to estimate, is the number of papers you got time a fixed factor that roughly takes into account the number of authors on the paper, the number of referees per submission, and the average number of submission. I would think that a factor between 1 and 3 is reasonable.
Then this reference could be mitigated in several ways.
For example if you struggle to stay active in research, and if lack of time is not the primary reason, then accepting more referee request is a good way to contribute; moreover it can help you get to grip again (see above and below).
Also, since there will be authors who will do less than their share (e.g. because they did a PhD and then quit Academia altogether, not necessarily free riders), each mathematician should do at least slightly more than his estimated share for the system not to get out of control.
So, if you don't feel like it and have some advance in your refereeing with regard to the above weighted reference, you have a good case declining. If you feel that you could do it and you are behind compared to the reference point, then accepting is the proper move community-wise.
1.b Accepting or not to referee a paper can be used as a political tool if you have strong opinion on how mathematical publication should be run. I think it is part of our duties to have a say in this. As a concrete example, you might decline to referee for some publishers whose practice are in your opinion opposed to the interest of the mathematical community (no need to be more explicit I guess, except it might concern more than one publisher). You might also decline to referee because the way you are asked is not correct; as an example if the editor asking me would imply the expected answer in his demand (e.g. "the paper is long and we have a large backlog so we would only accept it if it where really extraordinary"), I could very well only give a quick opinion and not referee it fully, as my time would be wasted if I dive into a paper whose fate is basically already decided.
On the contrary, you might accept in a borderline case in order to support a journal with particularly good practices (e.g. open access without author fees, low-cost subscription community-owned, journal working a new model you want to support, journal having an at least decent proportion of women in its board, etc.).
2. Reasons centered on oneself.
As mentioned in the second sentence of my answer, refereeing is not only a duty, it is also an opportunity.
2.a You can accept to referee because you expect to learn from the process. This is particularly suited for papers lying near the border of your area of expertise; in particular it might be the wrong move to only referee papers which are very close to your own work. Of course, a very technical and difficult paper needs more competence, so you have to weight that criterion against the expected effort and time needed to do the refereeing. If you feel a bit out of good questions to try answering, refereeing papers is a great way to force yourself into looking precisely at new math, which can only help find new directions. You may also find techniques which can help for questions you have no clue about, or it could turn out your own bag of tricks provides answers to some questions left open in the refereed paper.
2.b Accepting to referee and doing a good job at it will make you look good to the editor who solicited you. There is no short-term reward, but in the long run this can benefit you in several ways (being known to be serious could help your papers be taken seriously, at some point good refereeing job can help get proposed an editor position if your own scientific production is good enough). You should mention refereeing in your CV, but it will probably not make a big difference in most cases your CV is needed.
2.c Of course, the main reason to decline to referee is lack of time. You should not jeopardize your own career by accepting all referee request if those are numerous (I personally don't get so many that this argument weights much, but that depends on people and on subfields). If you feel guilty, look at 1.a and you might realize that you already did your share. Big shots can also reasonably concentrate on other things than refereeing "common" papers, and would be more selective (and less available).
