Research paper with experimental results from a small amount of data

Question

I have written a research paper in image processing that I will be submitting to an IEEE conference. I have proposed a new method to detect 'X' in images. However, due to hardware failure I could acquire only 4 sets of images. Thus, I have applied my method only on 4 images and I have also compared the results of previous two methods in literature by applying them to the same 4 images. The results obtained by my method are much better.

My question is, is it OK if I submit the paper showing results only on 4 images? The previous two papers have demonstrated their method on around 10 images or less. Also, is it OK to quantitatively compare the results (accuracy in percent) with just four images?

Answer 1

The amount of experimental evidence that is needed in order to adequately demonstrate a result depends strongly on the current state of the art that you are comparing against.

In some cases, even a single instance will be interesting and publishable, as there is much knowledge that can be gleaned from it. See, for example, the discovery of homo naledi: even though specimens have, to the best of my knowledge, only been found in the one cave, they are nevertheless clearly of interest and scientifically significant. On the opposite end of the scale, if you want to assert that you have a better method of determining significant websites than PageRank, then it will likely be difficult to convince readers unless you apply to datasets of a similar scale to the "whole internet" scale that PageRank is routinely applied to.

Without knowing details about your exact image problem, it is difficult to know how much can be learned from a single image. In general, image processing is a very well developed field in which most things are applied to large data sets, so my zero-knowledge guess would be that you need more data (though since you say the prior papers have <10 images, it may otherwise). On the other hand, you might be doing something where four images are actually significant. Finally, you can always send it in anyway and see what feedback you get.