53

Upon reading many papers in my field, which is Mathematics. I have come across many complicated theorems and proofs in which long overwhelming computations have been done. While paying attention to the peer-review time, I have noticed that many of those papers didn't spend a long time as they should, which is a deduction I based upon the complication of the proofs present in the papers.

This made me ask the question. Do experienced peer-reviewers occasionally ignore verifying "the math", and rather focus on the validity of the techniques used in the proofs and numerical simulations (if applicable)? Since experienced peer-reviewers are very busy academics, I guess it wouldn't be possible for them to verify each point in the proof. However, by not verifying each point of the paper, including the details in the proofs, the peer-review validity becomes questionable.

9
  • 6
    Similar question (but less specific in field): Do reviewers go over the calculations (i.e do the calculations themselves) when reviewing
    – Anyon
    Commented Jan 17, 2023 at 21:46
  • 1
    This is also addressed here: academia.stackexchange.com/q/131532/19607
    – Kimball
    Commented Jan 18, 2023 at 0:52
  • 2
    The answer may be field dependent. Peer reviewers in a mathematics journal might be more careful, but I found that an appendix in an Earth observation journal where I presented a mathematical derivation was ignored by all co-authors and all peer reviewers.
    – gerrit
    Commented Jan 18, 2023 at 8:49
  • 2
    This is generally reviewer-dependent. The last time I submitted an article on classical mechanics one of the reviewers checked and verified 80% of the calculations, the other reviewer did not check any of them.
    – Tom
    Commented Jan 18, 2023 at 18:59
  • 2
    While paying attention to the peer-review time, I have noticed that many of those papers didn't spend a long time as they should – What’s do you consider sufficiently long here? Except for extremely short review times, review duration is mostly an indicator of when the referees had time for the paper, not how much time they actually spent reviewing. Also see: Is it possible to make a decision upon a manuscript just within a month? (I know that pure mathematics operates on longer time scales, but the argument doesn’t change.)
    – Wrzlprmft
    Commented Jan 19, 2023 at 8:14

4 Answers 4

98

Preliminary remark

What is considered a "complicated computation" can depend quite heavily on the field, and also on the individual mathematician. But even in fields or papers that don't rely so much on lenghty computations, there might still be lengthy technical arguments, to which the same question applies.

It think it is worthwhile to add the following point to the answers already given.

About checking the correctness of proofs and computations

In my experience many people who start out in mathematical research have a somewhat inaccurate impression of how more experienced people (well, many of them, I guess, though probably not all) tend to check the correctness of mathematical proofs:

Going through every single line of a proof and a computation is not only lenghty and cumbersome, it also tends to be extremely (and by this I mean really extremely) unreliable. Human minds are not computers and are highly succeptible to various types of mistakes - in particular if we don't produce an argument ourself, but just read an argument that somebody else has written down. Without a very good intuitive understanding of a proof, it is just too easy to believe that "every step follows from the previous step" - and to overlook a missing assumption in some step, or a wrongly applied theorem, or a plain sign error, or what not.

What is much more reliable (and, luckily, also much more efficient) are certain high-level checks: figure out the crux of an argument, see where a new idea enters the game, test a result against the common counterexamples which you know prevent related statements from being true, see what part of the argument overcomes, in the present situation, the phenomenon that occurs in all these counterexamples, and so on.

Personal experience

I tend to regularly find critical mathematical mistakes when refereeing papers (although I haven't counted the precise percentage where I found such a mistake) - but I don't remember a single instance where I found a mistake by checking every line in a proof, noticing at one step that "this line doesn't follow from the previous line". In most cases it goes much more like this:

"Ok, Theorem 1 claims this and that. Under the additional assumption A, which is not there, I know that it's true - let me see if I can remember how the argument goes... Ah yes, and for this particular part of the argument I would need assumption A. Now let's see how they do it without this assumption, because this feels like quite a strong claim.

[Start to read the proof, skip the first ten lines because they contain just the usual stuff which one always does to prove results of this type.]

Ah, here's where it get's interesting. They say that claim C is true. Hmm, well, I can see that claim C, once established, will probably imply the theorem. Now let's see how they can get C without A. Ah, ok, here's the crucial argument. Hmm, but I don't understand why this should be true. Let me think again - ok, I have an example which shows that the crucial argument does not work, in general."

This can happen in many variations - but the point is always that, with increasing experience, one often does not need to check every line since one knows how the standard arguments in the field go. So instead one looks for the crucial parts of arguments where things happen that seem to be surprising or non-obvious (to somebody familiar with the standard techniques in the field).

Related

Closely related, and certainly worth a read: Terry Tao's blog post about the three stages of rigour in mathematics.

7
  • 13
    This is an excellent description of how a reviewer tracking down an error in a mathematical argument actually works in practice.
    – Buzz
    Commented Jan 18, 2023 at 0:40
  • 6
    Heh. I run into this all the time at work and not in mathematics. Someone can describe to you why their machine or technology, which fundamentally shouldn't work from first principles, in fact does. They will walk you through their reasoning and it will sound like it makes sense and it's really difficult to actually refute it from that perspective...except you know that fundamentally it shouldn't work. It's much easier to approach it from a higher level, less nitty gritty view independent of the author's reasoning and then it's usually pretty easy to explain exactly why it doesn't work.
    – DKNguyen
    Commented Jan 18, 2023 at 0:46
  • 1
    It's a lot like explaining Ramanujan summation, to be honest.
    – DKNguyen
    Commented Jan 18, 2023 at 0:52
  • @DKNguyen Ha, I happened to have done that yesterday here. Thing is that if a result looks to be a universal result then it cannot depend on the very specific details of any particular method to obtain it. Commented Jan 18, 2023 at 1:06
  • 2
    As a metaphor: "It's like getting driving directions, and the review is to see if one would be able to get to their destination using the given directions. But we usually don't actually go out and drive to full route." And the "Personal Experience" story corresponds to "Wait, there's no off-ramp from I-95 to route 20! Gotta look closer at that part.".
    – JonathanZ
    Commented Jan 19, 2023 at 18:41
23

Here is my perspective as a frequent reviewer in several areas of pure mathematics. I cannot speak for other people so I’ll answer the title question as if it’s asking about my specific practices:

Do peer-reviewers you ignore details in complicated mathematical computations and theorems?

Utopian short answer: no

Realistic short answer: sometimes, but only to the minimal extent necessary

Detailed answer: I consider it my responsibility as a reviewer to check correctness of the results. Correctness comes before anything else: if the paper is not correct it shouldn’t get published, period, and peer review is precisely the sort of adversarial process designed to minimize the risk that authors will get away with mistakes, sloppiness and outright dishonesty (all of which are common human behaviors). Not even trying to check for correctness simply defeats the purpose of peer review.

The only exception to the above is if a journal asks me to review a paper and explicitly says I don’t need to check correctness. And that has only ever happened to me in requests for a “quick opinion”, where I knew that if my opinion was favorable then a more formal and detailed referee report would get solicited later, and that report would include checking for correctness. If a journal asked me for a normal referee report but said I don’t need to check correctness, I would frankly worry whether such a journal should exist and what the point of it is. It would almost certainly devalue my opinion of the journal and of any paper published in it. But as I said, this is not something I’ve ever experienced.

That being said, there are utopian ideals and there is reality. In reality, it is not always practical for authors to include all the formal details of their arguments and calculations (doing so might inflate, say, a 60 page paper to twice the length, ensuring that it wouldn’t be read even by the very small number of people who would be willing to read through a very technical 60 page paper). And similarly it is not always practical for a reviewer to verify all the authors’ calculations - certainly the ones that were omitted, and even the ones that were included. So the only way for me to maintain the ideals I was espousing above would be to simply refuse to review papers where I knew I could not check all the details in a reasonable amount of time.

However, refusing to review a paper could carry its own negative consequences, since many review requests are in specialized areas where I am one of only a few people (or in some cases literally the only person) with the relevant expertise to be able to evaluate the paper. So if I refuse the assignment, it would fall to someone less qualified, who would still likely not check all the details, or worse, the paper couldn’t get reviewed at all. So the outcome of refusing to compromise my ideals could be worse than if I accepted to do the review.

The upshot of this analysis is that sometimes ideals must be compromised a bit. So what I end up doing in such situations (and what I would guess all serious mathematicians do when reviewing papers for respectable journals) is not check all of the details, but check some of the details — as many as can be checked in a reasonable amount of time — and beyond that, I try to look at the broader picture of the arguments the authors are making, see if they make sense by applying my intuition and experience with similar analyses, and running various “sanity checks” to see if the results seem plausible. It is a kind of probabilistic correctness checking, which can still work pretty well (I have caught many mistakes following this approach). With the level of experience I have doing this, I would like to believe that the probability of a serious error managing to get past me is fairly small. Although it is not zero obviously (in fact let’s be honest, it’s not zero even when I do check all the details).

Bottom line, my recommendation to any researchers reading this is, do not allow yourself to be sloppy in writing your papers out of a belief that you will get away with it because the referee doesn’t have time to check what you did. That would be doing a great disservice both to yourself and to your research area. Moreover, it is bound to catch up with you sooner or later.

1
  • 1
    This seems accurate. I would add that papers often omit details ("the proof is a routine diagram chase" for example) and I check some of these. Not all. Life is finite. Commented Jan 18, 2023 at 20:46
19

Indeed, in the last few decades, there has been a clearer mandate from editors to reviewers, to not check "correctness", but, more about "suitability for the journal".

This does pointedly suggest that refereed-journal-publication is not so much a certification of correctness, but a certification of having passed some status-gate-keeping.

Nevertheless, if/when the result is unsurprising, expected, and (therefore?) probably correct, no one cares too much about checking the details. No controversy is being opened. And, in fact, maybe no one really cares...

If/when a paper makes a big claim, then people do start looking critically at the details.

11
  • 19
    I can't tell you how sad this make me. It seems like a slippery slope to a garbage bin.
    – Buffy
    Commented Jan 17, 2023 at 21:55
  • 9
    @Buffy, yes, I agree... The commodification of "research" in recent years contributes to this. As well as corporatization of universities... Selfishly, I'm glad I just squeaked-by into tenure before things became more ridiculous. Hard enough to do good math without contrived sociological obstacles. Commented Jan 17, 2023 at 22:25
  • 7
    ... and also the saturation of the academic job market... with all of this veering off into the appearance of an absurd Ponzi scheme... Sigh. Commented Jan 17, 2023 at 22:28
  • 4
    Really? I have never had any suggestion from an editor of a math journal that I don't need to check correctness as a reviewer. (I have had such a suggestion, but only when reviewing for conferences.) Commented Jan 18, 2023 at 14:55
  • 3
    I have certainly had instructions along the lines of "Note that it is not sufficient for the results to be new and correct", but being a mathematician I am not going to confuse that with "it is not necessary"! Commented Jan 18, 2023 at 14:56
4

It's a bit of both. First, peer review explicitly states that the validity of the statements is not being checked, and is the responsibility of the author. Peer review decides if the results are original, significant and interesting, that is all.

Having said that, many referees do check arguments. For complicated computations, maybe not every line is checked, but whether it passes the 'sniff test'. Does every line look like it follows from the previous line?

Some referees will check every detail, some will be more holistic about it.

Edit: This was mentioned in the comments, but it deserves to be in the answer, especially as for some reason it was accepted over the other, better, answers. I said that referees are not checking validity. By this I mean that a mathematical theorem is either true or false, and referees do not, and realistically could not, check every paper in sufficient detail to confirm validity. And even then that just means two people have checked it, not that it is true.

Referees are certainly asked to comment on whether they think that a paper is correct. This is a very different thing: I can make a guess as to whether the paper is broadly correct, but I might well be wrong. It is down to the individual where on the spectrum, between checking every detail and just looking over the proofs, they lie.

But very few papers over, say, 50 pages long, have had every detail checked. One of my more recent papers (among other things) corrects the main theorems of three other major papers. Two of these three papers were peer reviewed, but the errors were not caught. This is neither the authors' nor the referees' fault, as the errors were subtle and the result could not be verified computationally at the time. It was only by approaching the problem in a completely different way that the errors came to light.

Do I believe the main theorem of this paper? Of course, but then so did the authors of the other papers. Do I believe my paper is error-free? No. I doubt if any paper over 50 pages long is error-free, because humans are not error-free.

14
  • 67
    "peer review explicitly states that the validity of the statements is not being checked": really? So if I submit a purported proof of the Riemann hypothesis to the Annals of Mathematics, they won't check it for correctness, but will only base their decision on whether the Riemann hypothesis is original, significant and interesting? Sorry but I think you're mistaken about that, even in much less extreme situations. Can you provide a source showing that peer review "explicitly states" what you claim it does?
    – Dan Romik
    Commented Jan 17, 2023 at 22:33
  • 25
    This sounds wrong to me; peer review typically check correctness of results/derivations.
    – Ben
    Commented Jan 17, 2023 at 22:59
  • 21
    Where does peer review explicitly state this? Commented Jan 18, 2023 at 0:47
  • 33
    What would it even mean for peer review to explicitly state this?
    – N. Virgo
    Commented Jan 18, 2023 at 4:53
  • 11
    @JochenGlueck there is room for a difference of opinions on the precise level of effort one can reasonably expect a reviewer to invest in verifying correctness, and on whether it is the reviewer or author who bears ultimate responsibility for the paper being correct. However, OP’s statement that "peer review explicitly states that the validity of the statements is not being checked” sounds factually false to me, and not a matter of opinion.
    – Dan Romik
    Commented Jan 18, 2023 at 8:22

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .