How to deal with consequential rounding errors when verifying the works of others?

Question

to give a stupid example of a rounding error: on a test, the student is asked to calculate the circumference of a circle with radius 6.2. You personally did the math with π rounded to 3.14 and calculated 38.936, but one of your students rounded to 3.14159, and because of that their result is 38.955716.

Now, I doubt any decent teacher would discard this result as wrong, since the math itself was correct. However, what if this calculation was part of a much larger series of calculations for an academic paper, and the resulting rounding error throws off the entire result? It's easy to just come out and say "if you round X to 15 places instead of 14, your formula makes the English channel 10 meters wider" or "you rounded to 5 places, but if you round Y to 6 places, your result is no longer statistically significant", but it just seems like such a cheap way to give feedback for academics. Another example: "by rounding X to Y+1 places (or Y-1 places) in our formula to determine whether an event happened, we managed to shift 90% of our test group from 'happens 25% of the attempts' to 'happens 75% of the attempts'". It's an extreme example, but something like that

I know that such a remark can be seen as a meaningful one since it's a viable concern, but it's still quite a nitpick, and there's probably a reason why X was rounded to 14 places. Still, I can't help but think that a formula should work regardless of how many places you round X to.

If an academic paper you are reviewing could change result or even be invalidated due to rounding differently, how should that be handled in a peer review?

Answer 1

Now, I doubt any decent teacher would discard this result as wrong

I'm generally considered a decent teacher, and, yes, I'd consider that result partly wrong and I'd remove a mark or two, because students, especially from certain fields (e.g. engineering), should learn to get the values right, also, not only the math.

There are mathematical tools that allow to evaluate the effect of rounding errors and to deal with uncertainty of parameters in general: if in an academic paper, the authors overlooked the effect of the rounding errors (getting the wrong result or suggesting a model excessively sensitive to the parameters), I'd suggest to reject the paper asking for major revisions, pointing out at the proper literature.

Answer 2

1. The underlying measurement precision is a key element of the rounding of any calculated number. In your circumference example, the radius is stated as "6.2" units (and not "6.20" or "6.200"). The sensible convention is to use only one more decimal place in the calculated answer as the least precise underpinning measurement. So, the circumference should be 38.96 units. Any other answer should lose marks, (assuming this is secondary/high school or beyond!).
2. It would often be difficult to establish that authors had not followed the normal conventions of rounding, though stating results to unreasonable numbers of decimal places, given the reviewers knowledge of the likely measurement error, would be sufficient reason to suggest the paper be revised.

Answer 3

When reviewing a paper, one keeps an ear tuned for red flags. Mishandling of precision issues could be a red flag. It could go either way (too little precision or too much). Frankly, if the authors use way too many significant digits for the context, that might catch a reviewer's attention most easily. Once a reviewer has noticed a red flag, then the fine-tooth comb comes into play and one starts checking the work at a greater level of detail. This is not desirable from the author's point of view, and it makes the reviewing work much more time consuming.

(The researcher will have chosen an appropriate level of precision, i.e. number of significant digits, based on an error analysis. But this is background work, not explicitly presented in the paper.)

If the authors did not use enough precision when doing the work, then what happens? They'll run into trouble before getting to the writing up stage. Calculations lead to other calculations, etc., and rounding errors build, to the point that the final result is meaningless and then there's no paper to write.

By the way, there is no need to keep this in the purely hypothetical realm. There's no reason a non-researcher with a reasonable math background can't dip a toe in the water and read some journal articles. A nice way to get started with this is to take a look at Nature. When you find an article that piques your interest, then you can take a look at some of the works cited. You may find that there are some topics and techniques you aren't familiar with, but you can look into any you feel interested in.