How to deal with insignificant work / wrong methodology in papers, when it is discussed correctly?

Question

I had a strange situation a few months ago:

I was reviewing a paper which was on an interesting problem and the authors used machine learning to solve this problem. Unfortunately, they seemed to have little experience with the methods so they obviously ran into the problem of overfitting without noticing this. Therefore, they drew wrong conclusions from their experiments. (Short summary for people not familiar with machine learning: They did not use the methods correctly and so their results were invalid).

I elaborated on the problem in my review and stated that the result cannot be generalized due to this problem and that the methods were used wrongly.

I was a bit surprised to receive a revision of the manuscript, where my remarks were just included in the discussion section: In fact they did not improve the experiment, but in the discussion they more or less stated that:

it is possible that everything we wrote in here is invalid due to the problem of overfitting.

So in the end they had a paper with a good justification for the project, a maybe working method which was not applied correctly, and they interpreted the results correct – so all in all it was a well documented example for a failure, and it was formally correct.

Still, my recommendation was to refuse the paper, but I was considering if it might be helpful to others because they could learn from the mistakes. On the other hand, publishing such a paper would not be a reference for the authors since this is a severe flaw in the use of methods (and readers might skip the discussion and think the results are valid).

Did you have similar experiences before? And how did you react?

Answer 1

In the end the decision on these sorts of things lies with the editor, not the reviewer. As reviewer you could write back saying

The authors now acknowledge the possibility of overfitting. In the abscence of any attempt test for overfitting (which I believe to be likely here rather than just a technical possibility) it is unclear what value the results obtained might have for the field, other than as a warning to others.

And leave it for the editor to decide.

Answer 2

I haven't personally had similar experiences before, and probably am much more junior than you (only really have reviewed a relatively small number of papers at all still), but here's my thoughts (I am familiar with the specific case of Machine Learning though):

I was a bit surprised to recieve a revision of the manuscript, where my remarks were just included in the discussion section: In fact they did not improve the experiment, but in the discussion they more or less stated "it is possible that everything we wrote in here is invalid due to the problem of overfitting".

Did they really only just add that in a discussion, or did they also adjust other places? Typically, a very short (e.g. one-sentence) summary of results is included in places like Abstract / Introduction / Conclusions, something like "our approach X performs well on problem Y". If there are any claims in any places like that, those would be unjustified in my opinion, and that would be a serious problem. There really shouldn't be any unjustified claims about performance anywhere in the paper.

So in the end they had a paper with a good justification for the project, a maybe working method which was not applied correctly, and they interpreted the results correct - so all in all it was an well documented example for a failure, and it was formally correct.

If the intuition / ideas behind the proposed approach are sound / interesting and new, if it all sounds like it could work out and be interesting... I think there might be some value in publication. Of course this also depends on the venue, some have lower standards than others. I suppose I'd treat it as a paper that simply didn't have any empirical evaluation at all, try to decide whether or not you'd still accept it then?

Still, my recommendation was to refuse the paper, but I was considering if it might be helpful to others because they could learn from the mistakes.

That's not what peer-reviewed papers are for. I do believe papers with "failures" in the sense of interesting ideas that, after a proper evaluation, turned out not to work should be valued more highly than they tend to be (only success stories that "beat state of the art" tend to be accepted). Such failures can provide useful insights when published, prevent others from coming up with the same ideas and wasting their time on them again. But accepting a paper with a methodological flaw so that others can use it as an example of what not to do in a paper? That doesn't seem valuable to me.