When a job posting asks for X research letters plus Y teaching letters, mathjobs still allows you to submit more than X + Y letters. (If mathjobs has an upper bound on the number of letters you can submit, I don't know it. I have seen files with at least ten letters, even though we ask for 3 + 1.)
In general, I think it is almost certainly okay to submit either the number of letters asked for or one more than that. For each additional letter that you want to submit, I would advise you to think about why you are doing it. Here are some things to remember:
It is not guaranteed that the hiring committee will read your entire application. If you submit, say, twice as many letters as asked for, there is a good chance that some of them will be read only very cursorily. This may not be what you want, since:
Virtually all letters I have ever read are "positive," i.e., they say good things about the candidate. But the impression created by sending Z letters is not the sum of the Z impressions created by sending each of the letters. Rather, unless each letter writer has qualitatively different very strong things to say about you, the total impression is some kind of weighted average of the letters, with the worst letters getting a higher weight than the best letters. Thus I think that for all but the most senior candidates it is probably a bad idea to submit more than four research letters: any more than that and it is very likely that the average impression would be raised by removing one of the letters. (Of course you still have the task of figuring out which are your best letters. But you should try to figure that out, not just admit defeat and submit way too many of them.)
In your case, it sounds like that they really are asking for 3+1 letters, and you want to send 4+1. That should be fine...but now is not too soon to try to figure out which of those four letters are the strongest and which are the weakest.
