I have only written but not reviewed such a paper yet. My personal perspective comes from the fields of complex systems and scientific computing, but I offer some generalisable ideas.
I structure my answer by the typical criteria applied to research papers:
This is arguably the most difficult-to-translate and most field-dependent aspect.
For example, in a simulational or experimental field, it is not feasible to completely verify the results of a research paper, as this would mean completely redoing it from scratch – all you can do (and are asked to do as a reviewer) is to check whether the employed methods are appropriate and the results are consistent.
On the other hand, in some theoretical fields, you can (and are expected to) check every calculation and proof.
I suggest to apply the same standard to software papers than to research papers and in particular method papers in the respective field.
After all, if a software paper gets accepted, it should be citable as “evidence” like a regular research paper and in particular, a methods paper:
With other words, citing the software paper should suffice as evidence that the respective task was done properly¹.
So, as a practical criterion:
- Consider a typical paper that would cite this software paper. Assume that the paper satisfies you with respect to scientific soundness otherwise.
- Suppose that the citation would be replaced with the content of the software paper.
- Would you still consider the paper scientifically sound (according to the standards of your field)?
More specific criteria would include:
Are the employed algorithms, libraries, etc. appropriately chosen, i.e., can they be hold to the same standard, do they fit the application, and are they state of the art?
Does the paper sufficiently back up claims made on the correctness, efficiency, and usability of the software? This point may slightly differ depending on the format of publication. For example, if the software itself is regarded as part of the publication: Can you verify these claims (possibly using provided tests, benchmarks, documentation, and examples)?
Does the software sufficiently adhere to good software-engineering practices such as tests that ensure that it works as intended?
¹ barring the correct handling of the software, which should of course be appropriately described by the citing paper
If the venue of publication has a novelty threshold, try to translate it to software.
Pragmatically, you ask yourself whether the paper in question will contribute positively to the journal’s impact factor (or some other metric).
Some specific criteria that come to mind:
- Can the new software perform any tasks that have not been automatised before?
- Is the software more efficient, more accurate, or easier to use than existing softwares by a relevant factor?
- Is the software a module for a programming language that did not have such a module before and that is relevant for scientists in the respective area²?
- How likely would you consider it that a user of an existing software performing the same tasks will switch to the software presented in the paper?
- Is the software sustainable, i.e.: Do you expect that the authors or somebody else will maintain the code in the years to come? Is the codebase maintainable and extendable or was it legacy code at the moment it was written? Does the software depend on some exotic libraries that may break in the near future³?
² Who would need a CAS library for Malboge?
³ This is an actual problem: One of my motivations to write a software for a specific task was that existing softwares either depended on discontinued and deprecated libraries or on a library that was heavily changed in a non-backwards-compatible way (which in turn was possible because hardly anybody used it).
Most other criteria should be easy to translate, in particular when you think of a method paper:
- Is the paper well written, using a reasonable structure, consistent terminology, and so on?
- Is it clear for which tasks the software can and cannot be applied?
- Are any shortcomings of the software (efficiency, scalability, problems with specific tasks) properly discussed?
- Are possible future expansions and improvements properly discussed?