I must admit that I tend to disagree with the previous answers: while the description of the specifics of the mathematical community are accurate, I do not see why this should affect the impact factor (except for the time needed before an article is cited, whose influence is clear). In particular, size of the field does not have any impact per itself on the average impact factor of papers. In fact, papers are less cited mainly because papers cite less.
Let me back my point, first assuming we are looking in a field that is closed (only cites itself and is only cited by itself) and stationary (no evolution of the number of papers published or the average number of references per article). Consider the publication graph of a given year : it is a bipartite graph, whose vertices are papers published year 0 (first partition) and years -1 and -2 (second partition) and whose edges are citations from the first ones to the second ones. Then the (article) average of impact factors AIF in this domain is the ratio
AIF = (#citations from year 0 papers to year -1 and -2 ones)/(#year -1 and -2 papers)
which is equal to (#edges)/(#papers published in two years), since the field is assumed to be stationary. This is also twice the average number of references to the two preceding years that a paper in the fields has.
So the article average impact factor of a closed and stationary field is solely governed by the references habits in the field. In particular, this is not affected by the overall size of the field (e.g. math as opposed to biology).
Given the distribution of references, an expending field will tend to have bigger impact factor, as will a field that is often cited by other ones. I do not feel that speed of expansion is an important factor for math compared to other fields, but fondamental mathematics are probably seldom cited outside itself. This has little impact if one consider maths against the rest of the world, though, since math papers seldomly cited outside the field too.
Another factor can be the distribution of papers among journals: for example, if a field has only two journals, one very large and one very small that only gets the very top articles, then the (unweighted) journal average IF will be extremely high. I doubt this explain much of the difference between math and the other fields, since mathematics have a strong hierarchy of journals.
So, what we really have to explain is why math papers cite less papers in the two-years range than papers in (most) other fields. This will explain why they are less cited.
Then the answer seems quite clear: maths papers are often long to read, and take time to be digested. The core of a biology paper is usually easy to understand and such papers are more easy to cite. There is also a small subfield effect: mathematician can work on problems that involve few previous papers. This is different from the size of fields, because it is more about the degree of specialization.