As in jakebeal's comment, if there is actual plagiarism, it will be picked up in the text, so you could use a generic plagiarism checker. Note the arXiv checks for text overlap and picks up quite a few things (the stated purpose is not to catch plagiarism, but it still does, though mostly "self-plagiarism" as far as I am aware).
That said, having done a fair amount of reviewing in math, this is not something that's typically done. Ideally, as a reviewer, you should be fairly aware of what's already been done in the area, and at the least, literate enough in the area to search for what has been done. (One tip: look at the crucial references, and check for papers citing them.)
What is much more likely than outright plagiarism is that some or all of the main results are already published, but unknown to the author. This may be because the results are not stated in the exact form the author is proving, or just the author wasn't aware of the appropriate papers. (I once pointed out that one of the papers the author referenced contained stronger results than the author proved--the author just didn't realize it!)
In any case, a proper literature search in the process of reviewing will turn up plagiarism almost all the time if it happens, and moreover is important for a good review. So my advice is don't worry about using a plagiarism checker, just do your homework.