How to review a software-tool paper?

Question

I often get asked to review papers on new software tools. This always makes me wonder how one should review such a paper, or even whether software papers should undergo a typical review process at all?

My private checklist typically consists of:

• Is it available for download for others?
• Does it run?
• Did the authors provide benchmark problems? Do they give expected results?
• Is documentation sufficient?
• Go through numerical methods used in the software (briefly, no meticulous code analysis)

If everything runs fine and produces expected results, I honestly have nothing else to “demand” from the authors. My referee reports are positive and (embarrassingly) brief in those cases. I’m of the opinion that the user community should “do the rest”; evaluate the software and decide whether it’s useful in their workflows.

Answer 1

One perspective is from the Journal of Open Source Software, which explicitly establishes reviewer guidelines that cover these questions (http://joss.theoj.org/about#reviewer_guidelines).

A (slightly edited) quote of the demands:

Software license There should be an OSI approved license included in the repository.[...]

Documentation There should be sufficient documentation for you, the reviewer to understand the core functionality of the software under review.

A statement of need The authors should clearly state what problems the software is designed to solve and who the target audience is.

Installation instructions There should be a clearly-stated list of dependencies. Ideally these should be handled with an automated package management solution.

• Good: A package management file such as a Gemfile or package.json or > equivalent
• OK: A list of dependencies to install
• Bad (not acceptable): Reliance on other software not listed by the authors

Example usage The authors should include examples of how to use the software (ideally to solve real-world analysis problems).

API documentation Reviewers should check that the software API is documented to a suitable level. This decision is left largely to the discretion of the reviewer and their experience of evaluating the software. [...]

Tests Authors are strongly encouraged to include an automated test suite covering the core functionality of their software.

• Good: An automated test suite hooked up to an external service such as Travis-CI or similar
• OK: Documented manual steps that can be followed to check the expected functionality of the software (e.g. a sample input file to assert behaviour)
• Bad (not acceptable): No way for you the reviewer to check whether the software works

Community guidelines There should be clear guidelines for third-parties wishing to: Contribute to the software, report issues or problems with the software, or seek support

This is a valuable way of thinking about reviewing software, and I think we should move closer to this. But it is not yet accepted! In my field, most "methods" papers don't even include software - and those that do would almost certainly fail these tests. (I'd never heard of continuous integration until recently, for instance.)

Answer 2

I have never reviewed an article but as someone that produces software (in bioinformatics) this is what I would like to know from a reviewer, expanding from your list:

• Is it downloadable and has a license?

  I modified your first point because the code should be available for others and should have a license to known what other users/developers can do with that tool.

• Does it work well? Do the authors made an effort to write the best tool?

• Could you install the tool in your settings?

• Are my tests enough to check that the software work as expected?

• What else could I do to ensure that it works well?

• Is the language of the tool a good choice? _{Is there a community of users and developers for that niche in that language?}

• Is it under version control? Is the tool under versioning?

The tools are usually developed when there is a problem:

• Does my tool solve correctly the problem targeted?
• Could this tool be used in other problems/situations I haven't thought of?
• Are there other methods to address the problem I am not aware of?
• Did I take into account all the relevant factors or I forgot something?

The tools are usually tested in a given dataset or compared to different tools as noted in a comment. _{This can be interesting if negative results are reported too.}

• Did I choose a good dataset ?
• Did I choose well the tools to compare my tool with?
• Is documentation sufficient?
• How easy is to understand how does the tool work?
• Does it clearly explain when you should use this tool?
• Does it provide enough examples or scenarios where this tool could be useful?
• Are the functions/methods/modules... easily understandable?
• Do I need to explain something better?

The software sometimes is developed by several people and several years:

• Is the tool documentation consistent? Did I forget to make the style and documentation coherent?
• Does it have a clear focus? Or did I went off at a tangent with other functions that don't bring anything to the tool and/or problem?

When using the software questions will come, you as a reviewer can check how will be handled:

• Does it have support from the authors? In a mailing list, a forum or alike? _{Preferably open to avoid having two users to have the same question}
• Is the long term support explained or thought?

As you noted some of these question might be addressed by the community but you as a reviewer have the opportunity to check that the article of that software explains and solve a problem for the community, that it is reasonable well done and well thought.

Answer 3

You might want to avoid reviewing trivialities and useless stuff. So, in addition to what Llopis said, ask whether the tool solves an interesting problem. Yes, this is a broadly and vaguely stated question, so feel free to adapt it to your area. For example, in formal methods, tools are the alpha and omega of progress, so, tool writers must get proper academic credit for proper work by, e.g., producing better results on standard benchmarks than the competitors.

Also, bear in mind that tool papers are usually different from research papers and should be evaluated differently, among themselves. Certain conferences have special pure-tools tracks. In such cases, it should be the task of the PC to provide the corresponding template for evaluation.

Answer 4

I have only written but not reviewed such a paper yet. My personal perspective comes from the fields of complex systems and scientific computing, but I offer some generalisable ideas.

I structure my answer by the typical criteria applied to research papers:

Soundness

This is arguably the most difficult-to-translate and most field-dependent aspect. For example, in a simulational or experimental field, it is not feasible to completely verify the results of a research paper, as this would mean completely redoing it from scratch – all you can do (and are asked to do as a reviewer) is to check whether the employed methods are appropriate and the results are consistent. On the other hand, in some theoretical fields, you can (and are expected to) check every calculation and proof.

I suggest to apply the same standard to software papers than to research papers and in particular method papers in the respective field. After all, if a software paper gets accepted, it should be citable as “evidence” like a regular research paper and in particular, a methods paper: With other words, citing the software paper should suffice as evidence that the respective task was done properly¹.

So, as a practical criterion:

1. Consider a typical paper that would cite this software paper. Assume that the paper satisfies you with respect to scientific soundness otherwise.
2. Suppose that the citation would be replaced with the content of the software paper.
3. Would you still consider the paper scientifically sound (according to the standards of your field)?

More specific criteria would include:

• Are the employed algorithms, libraries, etc. appropriately chosen, i.e., can they be hold to the same standard, do they fit the application, and are they state of the art?

• Does the paper sufficiently back up claims made on the correctness, efficiency, and usability of the software? This point may slightly differ depending on the format of publication. For example, if the software itself is regarded as part of the publication: Can you verify these claims (possibly using provided tests, benchmarks, documentation, and examples)?

• Does the software sufficiently adhere to good software-engineering practices such as tests that ensure that it works as intended?

---

^{¹ barring the correct handling of the software, which should of course be appropriately described by the citing paper}

Novelty

If the venue of publication has a novelty threshold, try to translate it to software. Pragmatically, you ask yourself whether the paper in question will contribute positively to the journal’s impact factor (or some other metric). Some specific criteria that come to mind:

• Can the new software perform any tasks that have not been automatised before?
• Is the software more efficient, more accurate, or easier to use than existing softwares by a relevant factor?
• Is the software a module for a programming language that did not have such a module before and that is relevant for scientists in the respective area²?
• How likely would you consider it that a user of an existing software performing the same tasks will switch to the software presented in the paper?
• Is the software sustainable, i.e.: Do you expect that the authors or somebody else will maintain the code in the years to come? Is the codebase maintainable and extendable or was it legacy code at the moment it was written? Does the software depend on some exotic libraries that may break in the near future³?

---

^{² Who would need a CAS library for Malboge?
³ This is an actual problem: One of my motivations to write a software for a specific task was that existing softwares either depended on discontinued and deprecated libraries or on a library that was heavily changed in a non-backwards-compatible way (which in turn was possible because hardly anybody used it).}

Other Criteria

Most other criteria should be easy to translate, in particular when you think of a method paper:

• Is the paper well written, using a reasonable structure, consistent terminology, and so on?
• Is it clear for which tasks the software can and cannot be applied?
• Are any shortcomings of the software (efficiency, scalability, problems with specific tasks) properly discussed?
• Are possible future expansions and improvements properly discussed?

Answer 5

The actual answer mainly depend on the specific journal/conference/workshop. My own experience is from High Performance Computing - there may be some field-dependent common practices, but I think most of it will be similar.

The main thing I see differently is that you are not reviewing the tool, you are reviewing the paper. Also a paper about a tool is not the same as documentation. Focus your review more on the scientific contribution of the paper rather than the tool itself. Ideally the paper itself should be useful without the tool. I would even argue that a paper describing a failed attempt of developing a new tool can be a valuable contribution for a slightly restricted audience.

There is no general limitation to open source tools (unless of course your venue is the Journal of Open Source Software). At least for HPC, paper about proprietary tools are not uncommon. However, publicly available tools do improve the reproducibility of a paper.

At the same time, any scientific paper should include a proper discussion of related work. The paper should describe tools that use similar techniques or address similar goals and how the new tool is different. If the tool leverages a published technique, it should be referenced. The related work can also include references that underline use cases.

A tool-paper doesn't need to produce an detailed description on how to use the tool, but should focus on the novel concepts applied in the tool, be it in terms of implementation aspects or features. Use cases are also an important part, but more with a focus to evaluate the tool (e.g. performance comparisons), and how it is useful rather than instructing the user how to use it.

While some workshops have a clear focus on tools, there are not only pure tools papers. It is possible to present a tool for X-analysis/optimization on a workshop on X. Or if a tool makes heavy use of a technique from Y, it would fit for a Y-conference. This shift in audience should reflect in the focus of the paper.