17

When writing a paper, there are inevitably mistakes. Many of these can be caught by careful revision, and the more time spend revising, the more mistakes which will be caught. However, clearly a draft should be submitted at some point.

How should one determine this cutoff point? How much does it vary by field, individual, etc?

As a concrete example, I recently submitted a paper to a big ML conference where one of our results was off by a factor of two. While we fortunately caught this error before the camera ready version, the experience has caused me to question whether I am spending sufficient time on revising my work before submission. I have heard that such errors repeatedly occurring in published work can seriously damage your reputation in math.

On the other hand, I wonder if the standard for errors like this differ across disciplines. For instance, if we had submitted the paper to a journal in applied math, then I believe such an error would have been caught by the referees since the review process there is more fine grained. Thus, it also seems possible that fields in CS, where conferences with short review processes (relative to math journals) are standard, have a lower threshold on what what kind of errors are acceptable to the community.

I've already discussed this with my advisor. However, I am also interested in hearing the thoughts of others on the issue.


Edit: Thanks everyone for the responses! I'd also be interested in the second part of the question about how these standards vary across disciplines, individuals, etc.

6
  • 10
    ML conferences may indeed be a special case, reviewing there tends to be superficial, and the number of completely wrong theorems (not just by a factor of 2) is depressing. Jul 25, 2021 at 8:50
  • Getting some other eyes on the manuscript is pretty essential, especially if you are starting out. Maybe the first person you could ask is supervisor, or friends who are mathematicians who can glance over the manuscript and spot obvious flaws.
    – Tom
    Jul 25, 2021 at 13:12
  • 4
    @MarcGlisse I am even more depressed by the general quality of the English text in ML-based papers.
    – P. Shark
    Jul 25, 2021 at 16:31
  • Do other areas of CS not have the same issues? Jul 26, 2021 at 16:15
  • 1
    @MarcGlisse well, often rather not even wrong. And yeah, the review process seem to be mostly about rating how exciting results a paper presents. Jul 27, 2021 at 9:38

6 Answers 6

23

Ideally, the desired error number is zero. But as you observe, it is difficult to achieve. One problem is that the author of a paper is probably the least able to proofread it. (Maybe not literally "least", but I hope you get the point.) The reason is that when reading your own work what you "see" on the page is too often what you thought your wrote, not what you did write. Your brain glosses over errors.

What you need to stamp out errors is a set of fresh eyes of people with enough background to find any errors. They aren't influenced by what you were thinking when you wrote the paper. This is one of the main benefits of the reviewer system, actually.

An important author in CS (Don Knuth) has the practice of funding bounties for errors found in his book. He will send a check to the first person to catch any error. (He is so renowned, however, that few cash the checks anymore.)

So, you do what you can, being as careful as you can, for as long as is reasonable. But get some new "eyes" on your work.


Note also that some important errors in important papers have gone uncaught for decades. I once found an error in a paper by a renowned mathematician about 40 years after the paper was published. I don't claim to be the first to notice, of course.

I'm working on a long term project and have, after a few versions, still seen some incredibly bone-headed errors. The biggest thing holding me back is the lack of those "other" eyes.

3
  • 1
    Thanks for your response, it makes a lot of sense. I'd be interested in further details about the difference between disciplines if you have any thoughts. In particular, about the difference in expectations for review. Jul 25, 2021 at 1:18
  • 2
  • In my field, the main journal's guidelines simply states that accepted submissions should be "as far as is reasonably practical, completely free from error". They still employ sub-editors, however, who do change things (grammatical, almost never mathematical) in the proofs.
    – Landak
    Jul 25, 2021 at 17:00
13

In addition to typos-versus-flawed-proofs, there is another distinction worth making, I think. Namely, some proofs really are (perhaps) delicate computations (or other intensely symbolic lines of reasoning), and thus their persuasiveness and believability is already somewhat fragile. For example, here typos are terrible.

In some contrast, some written arguments are more "narratives" of a procedure/process that could be carried out, with many details visibly dictated by "the plan". If/when the details are determined in a more top-down fashion (or can be made to seem so), the argument is much less sensitive to typos (or even larger blunders...)

I'm not claiming that there's a dichotomy of innate nature of arguments. Rather, I'm advocating more-narrative approaches to what might otherwise invite fairly fragile, long computations without a good accompanying narrative. More-narrative approaches can be more robust, and less sensitive to typos and other errors.

4
  • 9
    If you have a long fragile computation, please for the love of your favorite mathematical deity copy-paste it into a computer algebra system, substitute generic values for enough variables to make it tractable, and have the computer algebra system check every step. Chances are you've made a nonzero (but hopefully even) number of sign errors. Jul 25, 2021 at 5:38
  • 1
    @JoelReyesNoche: I think that "her " should be "here" (as we're talking typos... :-) ). Jul 25, 2021 at 8:59
  • 2
    @AlexanderWoo While I agree that CAS systems can be beneficial for spotting odd mistakes in calculations and programs alike, normally they do not provide a reliable way to “check” a calculation/program: www21.in.tum.de/~ballarin/publications/thesis.ps.gz. This is what the proof assistants are for (of course, nowadays there exist a variety of interfaces that allow combining the power of the traditional CAS systems and proof assistants). Jul 25, 2021 at 10:42
  • 3
    @AlexanderWoo When I make an even number of sign errors, I make them far enough apart that they fail to cancel out. Jul 25, 2021 at 18:41
10

Here is my approach (pure math):

  • I consider a paper ready for submission when I personally am completely convinced of it's correctness. If I only think, "it's probably okay", that's not good enough. That said, I do put more effort into verifying central statements than tangential asides.

  • When possible, I try to cross-check my results in different ways (do they contradict other approaches or heursitics) and compute examples. For me, computing examples and comparing with related results in the literature is very important for finding errors and convincing myself my finalized version is correct.

  • Explain the work to others, and ask others for feedback.

  • After I finish writing the paper, I put it aside for a couple of weeks, and then reread it as carefully as I have the energy to. If there were significant changes, repeat this. Do this again after referee reports.

This doesn't mean all of my papers are error free, but as far as I know there are no major errors in the final versions of my papers.

5
  • 1
    Typos: "it's correctness", "heursitics", "error free". Jul 26, 2021 at 15:11
  • 1
    @YuvalFilmus I don't try to make sure my SE posts are error-free!
    – Kimball
    Jul 26, 2021 at 16:01
  • My response to this, with an amount of seriousness safely bounded away from both 0 and 1, is: depending upon my mood / psychological / metaphysical state, I may not be completely convinced of the correctness of anything. (But, with over 50 published papers, I only know of one result that is really faulty: the proof is correct but doesn't prove as strong a result as claimed in the statement.) Jul 26, 2021 at 16:31
  • 1
    @PeteL.Clark I agree that it's subjective, and "completely convinced" might be a slight hyperbole, but I think the main thing I was trying to add to the existing answers was the computational aspect. I realize I am error-prone, and actually checking sufficiently diverse examples (when possible) is often what convinces me that the results are okay.
    – Kimball
    Jul 26, 2021 at 21:02
  • +1 for sanity checking with other people's results, examples, test cases, comparisons against earlier versions, looking at intermediate results etc. and taking as much time as it takes. Especially in complex algorithmic analyses it's so easy to incorporate mistakes. Please don't add more irreproducible papers with loads of mistakes in the details to the already huge pile.
    – jhin
    Jul 26, 2021 at 21:23
6

Depends on mistakes. If these are crucial mistakes in arguments, you should correct them before submitting the paper. It is doable. Misprints, on the other hand, are not that important and you do not need to catch them all. So to submit your paper you need to be absolutely sure that the arguments are correct. Recently I edited a paper submitted to "my" journal. The referee found that all proofs are correct. Then I found lots of small misprints and gave the authors a few weeks to proofread the text. When they do that, I intend to recommend the paper for acceptance.

1
  • 2
    Well, some mathematical misprints can be troublesome...
    – Kimball
    Jul 25, 2021 at 15:02
4

Assume the perspective of a Reviewer.

As a Reviewer, I will never reject a paper because of "mere" typos unless:

  • they make the paper unreadable. But this happens when the author(s) spend no time in spellchecking, or have very little English writing expertise (and both of these are different problems)

  • they are still widely present after the first round of review. I can tolerate some typos in the first submission, but my acceptance level decreases for subsequent rounds, especially if I made it clear that the paper required revisions in the English text (but this is also a different circumstance than what you are likely referring to)

  • the typo affects a crucial claim. These are the most crucial offenders, because these "mistakes" undermine the credibility of the paper and can be the ground for rejection. I once rejected a paper because the authors stated "X" as the main contribution of the paper in the Introduction, and then stated "not X" as the main contribution of the paper in Section 2; the rest of the text did not allow me to determine which was right, and most of the remaining paper was too complex for my expertise, so I suggested a rejection. Note that this was explicitly due to the usage of the word "not", which can single handedly flip the table around.

  • the typo affects Proofs. This is self explanatory and is strictly related to the previous point, but it demanded a dedicated entry.

  • the typo affects the Results (numbers, tables, figures). The result section is arguably (depending on the field) very important and usually the source of many significant mistakes - which may or may not be considered as typos. Consider a 2d plot where you have two horizontal lines, a red at y=3 and a blue at y=6: if in the text you say that "the blue line is clearly lower" that is a big problem. Was the mistake due to (i) the incorrect use of the "black/red" color, (ii) incorrect use of the "lower/higher" term, or (iii) maybe it was a problem in the plot? As a Reviewer I do not know, and hence I can use this issue to reject the paper.

In summary, my recommendation is to try to point out the key areas in your paper that are likely to be read by Reviewers and focus on having those completely free of mistakes. Perfection is difficult to guarantee, so at least focus on the big picture.

As a final suggestion, make the Abstract error free: abstract with typos are a red flag.

3
  • When making the abstract error free, also check the title (and author names)
    – user53923
    Jul 25, 2021 at 19:53
  • s/redable/readable? Ironic given the context Jul 25, 2021 at 20:24
  • 2
    @D. Ben Knoble, that's another suggestion: don't write a paper with your smartphone.
    – P. Shark
    Jul 25, 2021 at 21:52
3

Perspective of a recent PhD student here:

You learn to make papers "better" by writing papers. And by "better" here I mean useful to your audience. The goal of academic writing is not to create perfect piece of text that will stay valid forever, but to help others and to move the scientific discussion forward. You can do it even when you have mistakes and typos.

By writing and submitting more work you will improve your work process (for me it was learning to slow down and letting papers "marinate" for a few weeks without me working on them). You will also see your mistakes published, which will remind you to try harder next time.

However, typos (not critical logic/math mistakes) and other blemishes are not disqualifiers for an academic piece to be useful.

1
  • Definitely agree with marinating. Forgetting what you wrote, and then reading what you actually wrote, versus what you thought you wrote, is a good method of making a better paper.
    – masher
    Jul 28, 2021 at 6:43

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .