What should I do if a paper's theoretical justification isn't right, but the method based on that justification works?

Question

This is a problem that comes up every now and then when I am asked to review a paper.

In my old field of research (robotics), papers straddled the line between creative problem solving and theoretically sound science. The general layout of all of the papers is to first provide a scientific reason that your method should work, and then present the method and any tweaks you had to make in order for the actual engineering problem to be solved.

Many times this led to a dubious scientific background and leaps of logic.

Here is an example: A paper might be about an algorithm to solve X problem. X is something that humans do naturally (for instance, grasping a delicate object without breaking it), so logically one would go to the scientific literature to find out how humans do it. The paper would then include a detailed and theoretically correct background section into the biology that governs human abilities. Then, the paper would include a big leap in logic that is not supported by the actual science, and that leap in logic would be used to justify their algorithm. Authors would state this leap as though it is an established scientific fact, when in fact the science is either unsettled or their conclusions are wrong about the biology.

The problem is that the algorithm does in fact work when tested on the engineering system (a robot, in our continuing example). It just doesn't actually have any significant resemblance to the biology that the authors claim it was based on. So really the issue isn't that the algorithm is a bad one or that the testing and data were fudged; it's that the justification of the design didn't follow from the evidence presented. If it weren't presented as though it was decided fact I wouldn't even think about it.

In addition, it is highly unlikely that, if I were to ask the authors to justify their results better, they would be able to do it—the science simply doesn't support it, but they tinkered with it enough that it works.

What would be an appropriate response here?

Answer 1

It's appropriate to respond just as you have here: criticize the unfounded motivation and ask that the authors present their paper without it. For example:

the authors suggest that humans and other great apes navigate diverse terrain using wheels, however the authors do not support this assertion with appropriate citations to the literature on human physiology, and it seems to conflict with evidence that apes in fact use legs. Therefore, while their horse-drawn wheeled cart does seem like a promising logistical solution, I recommend that they remove any suggestion that this solution is biologically motivated unless the authors can provide sufficient references for their assertions.

As a neuroscientist I must say I very much sympathize with this problem, and occur it regularly on other Stack Exchange sites as people confuse what is known and useful within artificial neural networks with what is known about biological brains and neural circuits. However, while it's understandable to encounter these issues among learners in a field, it's not something that should ever make it into published papers.

Answer 2

I would just ask the authors to rewrite that part to ensure the science is correct, and state the algorithm is inspired by the biological aspects of human abilities.

This is how researchers in the bio-inspired (nature inspired) meta-heuristics research areas approach the problem; FYI, these researchers design algorithms that mimic biological processes.