I'm currently reviewing a paper which deals with iris recognition, submitted to a top conference in the field of biometrics. The paper's main contribution is a recognition model that, while presented as novel, is in fact essentially a very simple convolutional neural network.
However, in the experimental section the method is compared to several state-of-the-art approaches (including deep CNNs) and while the state-of-the-art achieves 70-80% accuracy, the method from the paper gets close to 100% accuracy. In addition, the methods were tested in 2 scenarios, first with 50 identities, then with 100 to recognise from. All the state-of-the-art methods performed worse in the 100-identity case (as expected), however the paper's method actually saw a slight increase in the performance accuracy.
This to me smells fishy, to put it mildly. However, there are 2 considerations:
- Technically it is possible (however improbable my experience in the field says this is), that the simple approach just handles this specific problem that much better than the deeper convolutional networks.
- Even if the experimental work was devised improperly, this may not have been done with ill intent. Simple mistakes in the training/testing procedures (such as training the model on the testing data) could conceivably cause such a discrepancy, without malicious intent on the authors' part.
To put it bluntly, I intend to reject the paper, however I would like to do so without making either direct or implied (possibly unwarranted) accusations of misconduct on the part of the authors.
How I have currently addressed this particular issue in my review (listed next to other shortcomings of the paper):
In the experimental section it is unclear how the models were trained. How was the data split into training, validation, testing data? Did all the trained models (including the [redacted] model) use this same data split in their training? In my experience a (near-)100% accuracy is unlikely to be achieved, regardless of the chosen recognition method, when the dataset is difficult enough that state-of-the-art methods (like [redacted]) only get 70-80% accuracy. Oftentimes such unnaturally high accuracy points to a mistake in the training/testing process (such as testing on previously-seen data). The fact that the performance improves from the 50-category case to the 100-category case, rather than dropping off as it (understandably) does for the other approaches, makes this even stranger. I would suggest the authors either publicly release the training/testing data and the code used, or provide enough detail in the experimental section to make these results directly reproducible.
Is this an appropriate way to address such results? Should I confer with the conference editor on this and voice my concerns to them directly?
