Researchers propose software tools as part of their research work to gather some metrics about application performance. I intend to propose a tool with better features and capabilities. For comparison I am using publicly available tools of other researchers.
The difficulty I am facing is that other researcher tool crash for some applications of benchmarks. Different tool crash for different applications. So when reporting the comparison what should I do? List applications for which other researcher tool did not worked or try to debug other researcher tool and then report the comparison?
The benchmark applications used for testing the tools are large scientific applications. The tools are written using complex software frameworks. In short, debugging either of them will require substantial effort. Before committing I would like to have some advise.