What is the threshold for a software paper to be publishable?

Question

I've been asked to referee a paper about a software tool. I'm struggling with whether this particular work is sufficient to warrant publication. The journal I'm reviewing for doesn't have a "notability/novelty" criterion, but they do require that the work be a "unit of publication."

Let's call the thing I'm reviewing "Feature." Feature is part of a larger package, let's call it "Package." Feature is a GUI tool that takes the data that has already been calculated by Package and plots it using a widely-used plotting backend. It has some nice stuff exposing options in the plotting backend as GUI elements, and a few options related to the nature of the plot being made (scatter plot vs. heatmap, basically, plus some toggles based on domain-specific labels in the data).

Here are some facts that are shaping my view:

1. There is essentially no scientific logic in Feature. It needs to be able to read the files, optionally multiply by a weight, and then it has a few stored toggles in the data to change which data is presented. But it's mainly a very simple GUI to analyze visualize data from Package.
2. The entirety of Feature is less than 1400 lines of code, with almost half being the GUI-specific.
3. Looking at the code, (especially non-GUI) I suspect that I could cut about 300-400 lines of code -- the developers aren't using scientific software tools available, including reimplementing a function that's in a library they include.
4. Currently, a paper about Package version X.0 is under peer review (X>1); cited in the manuscript I'm reviewing. All authors of the paper on Feature are also authors of the paper on Package.
5. The entire author list for Package is essentially one research group -- this isn't a giant community-wide effort. And the paper on Package version X-1 was published only 2 years ago, so I'm surprised they're trying to get two more software papers out already.

The last two especially annoy me, because I feel like the purpose of scientific software papers are to make scientific software development citable, not to pad your paper count. The authors will already be getting the citations based on the other paper.

On the other side, I came to this with some bias that one of the authors has a tendancy to care more about quantity than quality with publications. So I'm not sure if my leaning toward reject is partially based on that, or on the facts on the ground. (Hence the desire for other opinions.)

EDIT: I forgot to say that Feature is not yet merged into the main branch of Package, but it currently in a separate branch in the same git repository.

Answer 1

There is no bright line threshold for such things and the answer should be the same as for any scientific paper, hinging on questions of novelty and extension of knowledge. If it doesn't have that, then it probably isn't a good candidate, though the standards of different journals vary widely.

But you seem to describe a small-if-any advance with little novelty and you also seem to have made up your mind. We can't help you with the judgement and you just need to take a risk and call it. Others may disagree with you, but that is always the case in reviewing.

I don't really understand your point 5 and might disagree with your emphasis on it. Why is work done within a group less valuable that work that crosses institutional lines? Lots of papers are done by one or a few people within a research group. However, I might agree more if you mean all of their cited papers come from those same people. That isn't necessarily a red flag, but might be.

Answer 2

I'm going to accept Buffy's answer, because even if it is not the "here's how to make it easy on you" answer I'd hoped for, it seems to be the generally-agreed correct answer.

That said, after a lot of thought (and considering feedback from here) I've come up with a list that pretty much satisfies me. I guess that everything after the first two points is more to check "is there novelty/notability here that you're overlooking because your gut reaction is that this isn't enough work to count as a paper?"

• Does this implement any non-trivial science? If the tool implements significant data generation or data analysis logic, then it should be considered. "Significant" is still a subjective term, but just reading an output format would not be significant.
• Is the software tool distributed independently? One important purpose of software papers is to make the community aware of plugins that otherwise wouldn't be noticed (lost in some dark corner of the internet). These may be relatively small (but useful) bits of code that would be overlooked without a journal to serve as an announcement forum. This criterion is probably only applicable to edge cases.
• Is the tool re-usable in many contexts? Even a very simple tool can be a valuable contribution if it is designed to facilitate interoperability. For example, a simple "universal translator" for domain-specific formats might be very small, but might be worth publishing.
• Is the approach novel? Even for journals that do not have a novelty requirement, the existence of a novel approach can be a consideration in whether a paper should be accepted.
• Does implementation of the tool require significant domain expertise? Packaging significant domain expertise in a way that makes it available to non-experts is a potential purpose of a scientific software publication.

For the fifth point, I'm imagining something like a GUI that prevents users from selecting incompatible options, where the incompatibilities may require domain expertise. I know in my field we have script-based data generation programs where users can accidentally select incompatible options and get nonsense as a result.

I'm not positive that this covers all cases, but it covers everything I can think of. And I feel better having used a rubric that I could apply to any software package.

Posting with the hope of community feedback, and so the next person in a similar situation can see what I did!