When doing comparative evaluation among different research tools should we debug others research tool?

Question

Researchers propose software tools as part of their research work to gather some metrics about application performance. I intend to propose a tool with better features and capabilities. For comparison I am using publicly available tools of other researchers.

The difficulty I am facing is that other researcher tool crash for some applications of benchmarks. Different tool crash for different applications. So when reporting the comparison what should I do? List applications for which other researcher tool did not worked or try to debug other researcher tool and then report the comparison?

The benchmark applications used for testing the tools are large scientific applications. The tools are written using complex software frameworks. In short, debugging either of them will require substantial effort. Before committing I would like to have some advise.

Answer 1

You could do that, but then you wouldn't be doing a comparative study anymore. You would, in effect, be comparing one thing you created with something else that you modified. It would bring your comparison into doubt even if it didn't invalidate it.

Report that your tool produces X and tool A produces no result for such cases. That is a valid comparison.

But improvement of the tools of others can also be a valuable contribution, though of a different sort. Decide what it is you are after.

You could also point out the flaws to the developers of the other tools and see if they wish to update. If so, you can compare with their new versions.

You can also, perhaps, enter into a tool-building collaboration with some of the others.