I and my team members worked on a machine learning paper to predict pass and fail on a standard academic test. The paper was submitted to a reputable journal. The first review came-back with a lot of encouragement but with few suggestions which made us change the methodology of our work especially with respect to the data - preprocessing. The changes were made the revised manuscript was submitted again. However, one thing which should have been debated before submitting was the accuracy of the results which was 100% for multiple evaluation metrics on each of the stratified folds. While the concern was raised within our team we did eventually submit it without rigorously making sure if everything was right or not.
Now the reviewer comments have come back and he has challenged the credibility of our work specifically pointing out that 100% accuracy looks too good to be true. We re-ran the models to evaluate how it behaves and found the results are quite different and not close to 100% percent. We have reviewed all the code and the steps we took to achieve the results but can-not find any flaw or mistake that may have resulted in 100% accuracy.
The current results which hover between 65 to 80% between folds look far more credible. However, I am not sure how to respond to the reviewer especially when we are not able to find our mistake that resulted in the 100% accuracy. We want to be absolutely honest and want to acknowledge that we committed a mistake in submitting the 100% accuracy and we should have been more critical of our work.
While as a group we want to be absolutely transparent with all the data files and results that we have obtained and share it with the reviewer. But I am a little concerned about how would the reviewer perceive such a big difference in results which change the conclusion or at least the interpretation.
Should we submit our responses and revise the manuscript or withdraw the paper?
Any help would be appreciated.