10

The question is clear:

Do reviewers go over the calculations (i.e do the calculations themselves) in the paper? For example, it can be the derivation of an equation, or some numerical calculations.

Edit:

What I am trying to understand is that if I am reading a paper and seeing some derivations, should I assume that someone (possibly reviewers) other than the authors went through those derivations in detail.

Edit 2:

The question can be narrowed down to STEM fields only.

10

4 Answers 4

20

This depends on the field, the paper, and the reviewer. If the result of a calculation seems to be "expected" by an experienced reviewer they may feel no need to go over the calculation in detail. An exception would be when it is a core result on which other things depend. Another exception is if the result surprises them in any way.

And of course, some reviewers are much more meticulous than others and some fields require a lot of care because things can get subtle.

But the author should, on the one hand, assume that they do, so that they don't get caught in error during review, but also assume that they don't so that they don't get caught in error later. (Confusing, I know).

Ultimately it is the responsibility of the author to be correct and accurate and some errors do slip through review and don't get caught for a long time, if ever. People (authors and reviewers) do make mistakes, of course, but the combination of care from both tends to catch most errors.

1
  • 7
    "But the author should, on the one hand, assume that they do, so that they don't get caught in error during review, but also assume that they don't so that they don't get caught in error later. (Confusing, I know). " I'd also add that a reader should assume that the reviewer has not checked the calcualtions if they are at all in doubt. May 20, 2020 at 10:46
4

There are two types of result I'd light to highlight:

  1. Numerical results that depend strongly on computational methods;
  2. Theoretical results expressed algebraically.

Besides, there is the significance of results (which obviously depend on the referee):

  1. Results seen as relevant
  2. Results seen as irrelevant

Irrelevant results will usually be under the microscope if and only if the referee is a true expert in the field, but almost always go unnoticed (if they get published at all). The referee will probably consider them to be unworthy of being correct or incorrect, since they are irrelevant and don't even deserve his/her time. The doubt, in this case, doesn't hurt.

Now, if the results are considered relevant by the referee, then they will be judged as either expected or surprising. Expected results will probably not receive too much attention, but surprising ones might raise alarm bells. If the referee is an expert in the field and the result is algebraic, it is likely that he/she will analyze the steps in the calculations rather thoroughly, and even attempt reproducing them if they look feasible. This a desirable situation of a fair report, that is, the paper has fallen on experienced hands and the report by this referee will probably be favorable and useful - or full of question marks with respect to what was done in the paper if the referee didn't understand the calculations, which should help enhance the paper's readability. An unfair report might occur if the referee is less of an expert, since he/she will not attempt to reproduce the calculations and might question the paper's results with less substance. It is not uncommon for a referee that is not an expert to be convinced by the author's arguments, even if he/she himself doesn't go over the calculations.

We now come to the final case of a relevant, surprising paper whose results rely heavily on computers. In the absurdly vast majority of the cases, these are never reproduced. The expert referee might have some previous calculations pointing in a certain direction and compare what the paper says with his/her calculations, and he/she will probably include this in his/her report. He/she may strongly disagree with the paper due to reasons based on his/her own research, and even recommend that the paper should not be accepted, but he/she will probably never try to redo what the paper does. Sometimes the expert referee will ask for further evidence of the correctness of the results by suggesting another test to be included in the paper, and this often helps a lot in giving the paper more power or finally proving the main result was wrong.

The point is that you should always check everything when submitting a paper, and that if your paper deals with very unexpected results, then your tests need to be especially stringent, and your writing as clear as possible. Ground-breaking papers are the most important ones with respect to creating new research directions, so they really need to be subject to a lot of criticism and replication attempts - but the referee will almost never be the one to try replicating them.

1
  • I don't quite believe in an expected/surprising dichotomy. There are lots of things that weren't specifically but also aren't terribly surprising. I have in mind various explicit factors in equations, which one has no particular intuition about and computational errors can be subtle.
    – Kimball
    May 20, 2020 at 23:26
4

It would be unusual for a reviewer to replicate numerical calculations in a paper, but in some cases they might do so, particularly if the supplementary materials include computer script allowing them to replicate the results easily. (If the author has not supplied the reviewers with the computer script for the calculations, they can hardly expect the reviewer to program it from scratch.) A reviewer might decide to try to reproduce numerical calculations if the numbers in the paper "smell funny", but often they will just assume that the authors have correctly implemented their calculations as described in the paper.

As to the derivation of equations, these would usually be checked if they are part of the work in the paper, or if they look unusual. Papers that involve mathematical proofs or derivations of equations are generally reviewed with scrutiny on these proofs/derivations, and a good reviewer will be able to identify if there is an error or a part that is unclear. Derivations of results are usually only included if they are either original work, or if they are useful in understanding the material in the paper, so in either case a good reviewer will check them.

Ideally, reviewers should give a comprehensive review of the paper, and they should also include a statement setting out any limitations on the scope of their review in their report. (This is particularly important in multidisciplinary papers where an individual reviewer may only review one aspect of the work in his or her specialty area.) Unfortunately, most reviewers to do not provide a statement about the scope of their review and so, in the absence of comments on it, you will not really be certain whether a reviewer has checked a mathematical derivation or numerical computation.

1
  • 1
    On the other hand if the script is supplied, the question becomes "do reviewers validate the script?", leading to very similar conclusions
    – Chris H
    May 20, 2020 at 11:06
2

The answers given before are quite good. I just want to add a few points based on my own experience that may be relevant.

The author of the question did not specify the field. It is safe to assume however that it is of scientific nature.

Let us begin with pure mathematics. Most of computations in pure mathematics are theoretical computations. Here you want to check thoroughly the computation. As you do the hard work, sometimes you realize the bigger picture sometimes you don't. This means that the theoretical result may be simplified, with painless or easier computations. But in general, you do not know yet. So you have to go through the computations. As a personal example, this happened to me when I read Bigelow's paper on the linearity of the classical braid group. This involved a certain detailed computation. Later on the result was understood on another level that greatly simplified the original computation.

Even in pure maths, it may happen that a computation is an example. The author wants to illustrate some of the theory he's building, or is making a certain point. Even in this case, as a reviewer, I immediately go over the example and try to process the computations, especially if the rest of the paper is very theoretical. Why ? Because examples are the flesh of the theory. Without examples, there is nothing to eat. Examples help understand what motivates the author and conversely every fine author should strive at giving illuminating examples that guide the reader. So if the example involves a computation, I will go over it.

If the computation is related to an aside comment, it is a different issue. But it is relative to the field. If a pure math paper numerically computes some CPU time to convey a rough impression of complexity, this is one thing. If it is a computer science paper, it's an altogether different issue.

A computation may mean very different things for a mathematician, a computer scientist or a physicist.

Ultimately, the decision is up to the reviewer's best judgment. And it should do justice to the paper submitted in the sense that the relevancy/motivation to go over the computation is directly related to the originality of the paper.

This is particularly relevant to cross-field literature or interdisciplinary paper. As a mathematician, I occasionally stumble across common errors or bias in papers in neurosciences. In general I want to be able to reproduce the computation just as my co-author wants to reproduce the experiment. But equally often, I do have to state that I can review only a few aspects of the paper, since I am not qualified in some other fields. With respect to this particular issue raised by interdisciplinary papers, Ben's last paragraph is sensible.

A final word: errors are common. But somehow, over time, big (and not so big) errors are corrected or neutralized. I mean most errors that have resisted the reviewer's critical reading are either inoffensive or disappear. This is not meant to imply blind reliance of the reviewers or peers. But you should not be excessively guilty either on relying on them, especially if their field of expertise is far from yours.

FDW

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .