Benchmarking my research results with related work's results

Question

I have an accepted paper in CS, in which I proposed a new method in data-science related topics (data-intensive analysis, a lot of hyperparameter tuning and design decisions).

The reviewers opinions are positive, the method is clearly described and the results are validated properly. Thus, the conclusions of the work are valid.

However, one of the reviewers wrote that the paper lacks an empirical comparison with other similar works.

From my point of view, comparing the result of my method to the results of the related work is not viable for many reasons, such as the problem I am tackling is slightly different, the type of the dataset I am using is also different.

So, I see a fundamental difference between my work the related work that makes any comparison really not valid, though both my work and the related work are in the same area.

Any suggestions to improve my paper?

Answer 1

[...] comparing the result of the my method to the results of the related work is not viable for many reasons, such as the problem I am tackling is slightly different, the type of the dataset I am using is also different.

The first instict when writing a paper is to emphasise the differences with existing work in order to demonstrate contribution. However, the paper still belongs to different parts of the literature. As you say, the problem is slightly different and using a novel dataset does not automatically imply a methodological contribution, which means that other people have explored the same or a similar question using different data (which allows a comparison) and that other people have used similar methods (which also allows a comparison). I think re-reading the literature will cover the requirement.

If you are afraid that showing such links will undermine the novelty of the paper's contributions, that is another story. For work that is truly innovative, there is no such fear. For work that relies on many small tweaks, it is situational and often depends upon the reviewer.

Answer 2

If the paper was accepted, it means that the reviewers and the editor / chair considered it to have sufficient merit for publication. Comments in the reviews are suggestions to improve the paper, but being the author, it is finally up to your discretion as to what is appropriate.

In your case, assuming that you now need to send the final camera ready version for publication, you could update the paper with the reasons you think such a comparison would be invalid. This would also help future readers of your paper who may have the same question.

Answer 3

Arrange a comparison

The implied argument there is that if you want to make a solid argument about the superiority of method X for task A, then it's best practice to compare it empirically with related work. Your argument that it's not possible to directly compare numbers of your method for your task with a different method for a different task is technically correct, but not really sufficient.

So in the scenario of you proposing method X for task A and related work using method Y for task B, it would be methodologically proper to arrange a comparison by using method X also for problem B, or method Y also for problem A, depending on what's the main novelty in your paper.

If it's the task (because it's an important task for some reason) then you would need to provide some evidence why a new method is needed at all. You'd be expected to first apply the known methods from related work to your task/dataset in practice, reproducing their approach even if you believe it can't possibly be good enough - so that you'd have evidence that they are not good enough and your method is necessary.

Alternatively, if you believe that your method is superior, then it would be considered appropriate to try it out also on the dataset(s) that related work was using, compare and contrast. If you pick a substantially different dataset, then if you run all your experiments only on that, then these experiments can not say much about how your method compares with others.