5

I and my team members worked on a machine learning paper to predict pass and fail on a standard academic test. The paper was submitted to a reputable journal. The first review came-back with a lot of encouragement but with few suggestions which made us change the methodology of our work especially with respect to the data - preprocessing. The changes were made the revised manuscript was submitted again. However, one thing which should have been debated before submitting was the accuracy of the results which was 100% for multiple evaluation metrics on each of the stratified folds. While the concern was raised within our team we did eventually submit it without rigorously making sure if everything was right or not.

Now the reviewer comments have come back and he has challenged the credibility of our work specifically pointing out that 100% accuracy looks too good to be true. We re-ran the models to evaluate how it behaves and found the results are quite different and not close to 100% percent. We have reviewed all the code and the steps we took to achieve the results but can-not find any flaw or mistake that may have resulted in 100% accuracy.

The current results which hover between 65 to 80% between folds look far more credible. However, I am not sure how to respond to the reviewer especially when we are not able to find our mistake that resulted in the 100% accuracy. We want to be absolutely honest and want to acknowledge that we committed a mistake in submitting the 100% accuracy and we should have been more critical of our work.

While as a group we want to be absolutely transparent with all the data files and results that we have obtained and share it with the reviewer. But I am a little concerned about how would the reviewer perceive such a big difference in results which change the conclusion or at least the interpretation.

Should we submit our responses and revise the manuscript or withdraw the paper?

Any help would be appreciated.

  • 1
    Are you able to reproduce the 100% accuracy results? Or did you simply rerun the code (without modifying it) and got a different accuracy? – Allure Jun 18 at 0:59
  • No, we are not able to reproduce the 100% accuracy. We simply re-run the code (without modifying it). – Saad Jun 18 at 7:08
  • 3
    This is going to sound harsher than I mean it. But it will help to put this into no uncertain terms: It is impossible that you are doing exactly the same things and getting different results. Something must be different. The fact you haven't figured out what that is proves that you do not sufficiently understood your method! – user2705196 Jun 18 at 15:43
  • 1
    I totally understand what you mean. To be brutally honest I really don't know what went wrong and this obviously raises concern that whether or not do I fully understand the methodology. My best guess is a technical glitch occurred while logging performance metrics but yeah I know this a tricky situation and we are doing our best to see what best respons we can comeup with and then decide whether to respond or withdraw. – Saad Jun 19 at 15:26
3

I suggest that you should withdraw your submission until you can figure out what's going on. Good science should be reproducible, and this is not. You did not appropriately critique your own results, and put forth a result that seemed to "look good". You've now changed some part of your modeling methodology, and gotten another set of results that "look good", even though you cannot articulate any theoretical or practical reason why this set of results is any more sound than the last. The first result did not pass the "smell test" of reasonable performance, and you kind of shot yourself in the foot by failing to investigate. Now you have a result that does pass the "smell test", but the reviewer is aware that your diligence in self-criticism is somewhat lacking.

You're essentially telling the reviewer that "the last result was an error, but this one is not, although we are unable to explain any difference between the two." It's a really big ask for the reviewer to have confidence in your updated methodology, if you yourself cannot explain why it is any more correct than what you did the first time around.

| improve this answer | |
  • The results are now reproducible the only thing I am worried about is what went wrong the last time. We have not changed any of our methods to get the results. We just re-ran the models and see if the behaviour of 100% persists or not. So the right thing would be to withdraw the paper if we cant find a good rationale of what went wrong the last time? – Saad Jun 17 at 20:02
  • 2
    I wouldn't withdraw the submission, but otherwise this answer puts it very nicely. You need to be extra diligent to ensure that the results are correct this time. Maybe add another test dataset, or an alternative implementation, to see if you get similar results? Basically, see if you can add something to assure both yourself and the reviewer that these results are true. I understand that it may not be possible now to trace what went wrong the first time; take it as a lesson to make your code more reproducible at every stage, and move on. – juod Jun 17 at 21:40
  • 1
    To be the Devil's advocate, the first result could be a glitch or might be based on a coding error which cannot be picked up. – user117109 Jun 17 at 22:16
  • @Titus my best guess is some sort of glitch occurred during the recording/logging of performance metrics but guesses won't work. – Saad Jun 18 at 7:09
  • 1
    @juod Yes you are right we need to be extra diligent. Its a lesson learned. – Saad Jun 18 at 7:11
1

You should definitely either withdraw or revise the manuscript in a way that explains what you did wrong earlier, what you have changed and what your new results are. Even if you pass the reviews and it is published, someone will question it and your credibility will be affected. 100% is indeed quite questionable to be fair, which largely suggests your training and test data are too similar / overfitting / that there is a data leak etc. which is a very fundamental mistake. Noone is going to bash you for correcting your mistake, even though you might be feeling ashamed.

I was in a group once and a labmate said they fixed the problem we had with overfitting without disclosing his edits in detail and our lab leader insisted we complete experiments and write the paper and send it out despite my suspicions and clear out-outspokenness about it. He was even mean to me about it when I questioned the labmate's magical solution. Later, finally by chance we found out the labmate changed the code to use training data as the test data, so he was training on the same data and testing on it too (we got 96%-98% accuracy even then - this is explainable in neural nets). He had already left the job as his contract was ending, and clearly this was no mistake. We found this after we wrote the whole paper about it and I have spent so much time on it, but I was GLAD we found it after all, it could likely pass the reviews but sooner or later my credibility would go down with that paper.

| improve this answer | |
0

You have a paper that you, yourself, describe as flawed. I suggest that you fix it before you move forward. Perhaps there is time to do that without withdrawing it, but if you push it forward only bad outcomes are likely. A "response" rather than a correction, is probably not enough.

The journal might reject it. But if they publish it, the readers might, well, question your methods.

If you have time to fix it while staying in the publication process then do so, but otherwise, it is probably best to withdraw it until you can find the errors.

| improve this answer | |
  • I won't say the entire paper is flawed. The results, however, should have been rigorously debated. The real issue is why 100% that time it may be due to technical glitch in recording metrics or something else but I can't find anything wrong in the process. We have re-run the experiments and have the results to be reported. We aren't going to push forward with the flawed results. – Saad Jun 17 at 19:36
  • 1
    Good, but if you don't know why you got the results you did, then you still have an issue. Is your work reproducible, for example? It is your reputation on the line, of course. – Buffy Jun 17 at 19:40
  • Exactly, so unless we find a good rationale to support what went wrong the last time we should withdraw the paper? – Saad Jun 17 at 20:02
  • I can't honestly say. But you need to know before it gets published. If you are getting inconsistent results you need to work that out. My preference would be to work hard on it and try to get a better version before a deadline requires a decision from you about withdrawal. If software is involved here, maybe you got a bad run from bad parameters. If it is a pure statistical model, then an anomaly is possible. Rare things do happen - and must eventually. – Buffy Jun 17 at 20:06
  • Now we are getting consistent results I don't know what went wrong the last time. Maybe it was a technical glitch or something similar. – Saad Jun 18 at 7:38
0

Just say that you have re-run the experiments, and made an error. Whether the paper is rejected depends on other parts of the paper. Your problem or solution could be novel, and hence your solution is the best result thus far. In that case, 60%-80% is ok. In fact, this might lead to many follow up works, i.e., citations, if your problem is interesting. However, if there is a better solution, then there is nothing to publish.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.