I had a strange situation a few months ago:
I was reviewing a paper which was on an interesting problem and the authors used machine learning to solve this problem. Unfortunately, they seemed to have little experience with the methods so they obviously ran into the problem of overfitting without noticing this. Therefore, they drew wrong conclusions from their experiments. (Short summary for people not familiar with machine learning: They did not use the methods correctly and so their results were invalid).
I elaborated on the problem in my review and stated that the result cannot be generalized due to this problem and that the methods were used wrongly.
I was a bit surprised to receive a revision of the manuscript, where my remarks were just included in the discussion section: In fact they did not improve the experiment, but in the discussion they more or less stated that:
it is possible that everything we wrote in here is invalid due to the problem of overfitting.
So in the end they had a paper with a good justification for the project, a maybe working method which was not applied correctly, and they interpreted the results correct – so all in all it was a well documented example for a failure, and it was formally correct.
Still, my recommendation was to refuse the paper, but I was considering if it might be helpful to others because they could learn from the mistakes. On the other hand, publishing such a paper would not be a reference for the authors since this is a severe flaw in the use of methods (and readers might skip the discussion and think the results are valid).
Did you have similar experiences before? And how did you react?