How does a math journal react to a crackpot math research paper? In mathematics (especially in Number Theory) there are many meaningless papers on the web, and possibly a ton of meaningless submissions. Also, how does a journal react when a proof of a major unsolved problems, such as the 'Riemann Hypothesis', is submitted?
Here are the basic mechanisms that I know of, having only limited experience on an editorial board.
The first is a classic desk rejection. This is where the editor who makes the initial assessment, just says no, without asking for referees. This has to happen, or one gets flooded with AI generated nonsense as a joke.
Some journals insist that you list a few potential referees. I believe that one of the reasons they do this is to see if you have interactions (even just closely reading their papers) with mathematicians. If the author of a number theory paper list as potential referees three Fields medalists from three different areas of mathematics, probably they have no idea what they are doing. Makes a desk reject easier.
The second is the editor asks for a quick opinion from a mathematician in vaguely the correct area, asking if the paper is worth refereeing. You can tell if you get this type of rejection sometimes as the editor says after a few weeks something like "the feedback I have gotten are that this is out of scope for our journal." So I have heard. From a friend.
Finally there is just sending it to one or more reviewers. If the editor needs one referee, asks four in succession and they all say no, then the editor is likely to reject the paper.
There are problems with these methods. Some innovative work is hard to classify, and a paper gets rejected because the editor asks all the wrong people. I had a paper go to a topic editor, go out for review, then get assigned to another topic editor, out for several reviews. It finally landed on the correct reviewer, but this could easily have gotten rejected.
A famous number theorist once told me he receives approximately one submission a week claiming to prove the Riemann Hypothesis at the top journal where he is an editor. Thus the need to conscientiously and in good faith evaluate such submissions is a real drain on his time and the time of other experts in the community. It’s not an acceptable solution to reject them all without even looking at them, and neither is it acceptable to handle them all using the “usual” process.
Fortunately, if I remember correctly what he told me, he can rule out all but an extremely small percentage of those submissions as being obviously invalid (where “obviously” means he doesn’t have to spend more than 5 minutes looking at them, and can then desk-reject them with an easy conscience and even sometimes provide a bit of feedback to the authors) by applying a simple set of heuristics he developed over time. They go something like:
the paper doesn’t use standard notation or terminology, or otherwise show some obvious misunderstanding of the problem or related body of knowledge
the paper uses only properties of the Riemann zeta function that are shared by a larger class of functions, for some of which the analogue of the Riemann hypothesis is known to be false
the paper proves something that’s actually much stronger than the Riemann hypothesis, and known to be false
[etc - there are probably more subtle rules of thumb that are still considered by him very trustworthy given his expertise in the subject]
I believe other experts who are regularly asked to evaluate purported proofs of famous open problems have developed similar heuristics. See for example Scott Aaronson’s “Ten Signs a Claimed Mathematical Breakthrough is Wrong”.
The bottom line is that it is actually quite difficult to come up with a crackpot paper that doesn’t instantly fail a set of obvious sanity checks that experts will know about but the crackpots (or honest amateur mathematicians who are not complete crackpots but simply lack training in the area) don’t. The only people who actually have the ability to “fake” a proof in a way that will require significant effort to detect the fake, are experts themselves, and they usually (though not always) know better and don’t write such papers in the first place.
Your question has a somehow detailed answer in the following paper:
W. Hodges, "An Editor Recalls Some Hopeless Papers", The Bulletin of Symbolic Logic, Vol. 4, No. 1, pp. 1-16, 1998.
That paper is dedicated to the many papers the journal received refuting Cantor's proof that real numbers and natural numbers have different cardinalities. It analyses the paralogisms contained in these papers, and I think that what it says can be generalised to other situations as well.
But beware that it's not only maths that receives crackpottish submissions: each field has its targets. For instance, in physics, relativity, quantum mechanics and Newton's third law are quite a target. And the submitters may be professionals and not only amateurs. To give you an example, there is a now retired well-known researcher in my field (metrology) who about twenty years ago developed a "theory" which he thinks should substitute Einstein's general relativity. So, about twenty years ago, he started to submit his work first to physics journals, which outright rejected it, and then to conferences in his own field, hoping for acceptance from his colleagues. This caused some headaches to the organizing committees, which anyway rejected his work (as far as I know, it's not been published so far).
So, as far as I know, most editors probably simply shrug those kind of submissions off.
I'm not a mathematician and I never saw this happen myself, but I heard from a friend that the policy of their department was that if a crackpot paper was received they gave it to a grad student to look over. The instructions to the grad student were "Read it until you find the first error, then STOP, write a quick summary of the error, and then send it back". I think the grad students took turns with this.
I don't remember why but apparently their department tended to be a magnet for papers about a particular topic (possibly the Riemann Hypothesis thing, maybe? I don't remember and am not a mathematician) so any paper coming in on that topic got the above treatment.
There is always the possibility that some gem is hidden in what otherwise seems like a crackpot submissions. An editor has to accept the that a field can be revolutionized by a single new idea.
A good editor knows most of the ways that thinking goes astray and can categorize wrong ideas easily and tell the author where their own thinking is wrong and respond with stock responses they've already collected. When they can't do this, either the individual is onto something new or a new path of wrong thinking has been found and the editor can update their set of premade responses. In either of these cases, the world gains.
If you can't do this, you've succumbed to the hubris that established power often falls into and it serves no one, ultimately. Honestly, unless it's vandalism or disrespect of the establishment, people should be worthy of responding to.
If we have a society where people are too worthless to give criticism or feedback, then a Doctor of Philosophy has to step up and understand what's going on in their society.