I submitted a paper in a major machine learning conference recently. The paper submission deadline passed ~2 weeks ago and the appendix submission deadline passed ~1 week ago. I just realized an issue in my testing pipeline that made the benchmark numbers incorrect. I reran the benchmarks and the numbers don't change the results (in the same "category"). What do I do (withdraw or request amendment)? I would not want the paper to get published with incorrect numbers (out of academic integrity), but it realistically should also not change much review wise.
It sounds like you are unaware that the paper, if accepted, will not be published in the verbatim form that you submitted to the conference, but there will be a camera ready version, where you are supposed to incorporate reviewer comments, and this will be the final version that will go into conference proceedings.
If these changes do not change the results and conclusions, then you can update the results with the latest ones and submit it as the camera ready version.
If the updated results do change conclusions, then the paper should be withdrawn as in most conferences you cannot update the paper after it has been submitted and the deadline passed.
As an example, for the Neural Processing Systems Conference (NeurIPS), this conference has a rebuttal process where authors can add new results in their rebuttal,in their 2023 FAQ they clearly specify this:
Can the rebuttal include new results? Yes, however your original submission will serve as the basis for the reviewers' (and ACs') acceptance recommendations. The rebuttals should serve only to clarify the reviewers' and ACs' questions during the discussion period.
And for the International Conference in Learning Representations (ICLR), you can update the paper during rebuttal, as long as you are clear to the reviewers of what has changed. This is very clear in the FAQ:
Q. Can we change the title of a paper during the rebuttal? Yes, you can change the title, abstract, and the paper’s content, including supplementary materials. But make sure any modifications are clearly communicated to the reviewers and the area chair, so that they can efficiently review the modified version of your paper. The set of authors cannot be changed, but the order can be changed.
The ethical thing to do is to withdraw the paper. We are scientists, and our goal is to tell the truth about the world as we see it. If you made a mistake, minor as it may be - own it and fess up. Here’s a few other reasons:
- It’s a slippery slope: if you let yourself go this time, you may make greater allowances the next time.
- It’s better if you own up than if a reviewer finds out/the numbers don’t make sense. Someone might ask to see your code for reproducibility. What will you tell them?
- One discovered error usually implies that other errors exist. Can you say for sure they don’t?
- ML is suffering from a serious reproducibility crisis, don’t contribute to it- be a force for good!