As in jakebeal's comment, if there is actual plagiarism, it will be picked up in the text, so you could use a generic plagiarism checker. Note the arXiv checks for text overlap and picks up quite a few things (the stated purpose is not to catch plagiarism, but it still does, though mostly "self-plagiarism" as far as I am aware).
That said, having done a fair amount of reviewing in math, this is not something that's typically done. Ideally, as a reviewer, you should be fairly aware of what's already been done in the area, and at the least, literate enough in the area to search for what has been done. (One tip: look at the crucial references, and check for papers citing them.)
Update (2024): I happened to look at a publisher's reviewer guidelines today, and they indicate the journal does use the iThenticate plagiarism detection system, so perhaps this and AI-detection tools are becoming more common in math. However, using such tools is still not a concern for individual reviewers. End update.
What is much more likely than outright plagiarism is that some or all of the main results are already published, but unknown to the author. This may be because the results are not stated in the exact form the author is proving, or just the author wasn't aware of the appropriate papers. (I once pointed out that one of the papers the author referenced contained stronger results than the author proved--the author just didn't realize it!)
In any case, a proper literature search in the process of reviewing will turn up plagiarism almost all the time if it happens, and moreover is important for a good review. So my advice is don't worry about using a plagiarism checker, just do your homework.