I've been asked to referee a paper about a software tool. I'm struggling with whether this particular work is sufficient to warrant publication. The journal I'm reviewing for doesn't have a "notability/novelty" criterion, but they do require that the work be a "unit of publication."

Let's call the thing I'm reviewing "Feature." Feature is part of a larger package, let's call it "Package." Feature is a GUI tool that takes the data that has already been calculated by Package and plots it using a widely-used plotting backend. It has some nice stuff exposing options in the plotting backend as GUI elements, and a few options related to the nature of the plot being made (scatter plot vs. heatmap, basically, plus some toggles based on domain-specific labels in the data).

Here are some facts that are shaping my view:

  1. There is essentially no scientific logic in Feature. It needs to be able to read the files, optionally multiply by a weight, and then it has a few stored toggles in the data to change which data is presented. But it's mainly a very simple GUI to analyze visualize data from Package.
  2. The entirety of Feature is less than 1400 lines of code, with almost half being the GUI-specific.
  3. Looking at the code, (especially non-GUI) I suspect that I could cut about 300-400 lines of code -- the developers aren't using scientific software tools available, including reimplementing a function that's in a library they include.
  4. Currently, a paper about Package version X.0 is under peer review (X>1); cited in the manuscript I'm reviewing. All authors of the paper on Feature are also authors of the paper on Package.
  5. The entire author list for Package is essentially one research group -- this isn't a giant community-wide effort. And the paper on Package version X-1 was published only 2 years ago, so I'm surprised they're trying to get two more software papers out already.

The last two especially annoy me, because I feel like the purpose of scientific software papers are to make scientific software development citable, not to pad your paper count. The authors will already be getting the citations based on the other paper.

On the other side, I came to this with some bias that one of the authors has a tendancy to care more about quantity than quality with publications. So I'm not sure if my leaning toward reject is partially based on that, or on the facts on the ground. (Hence the desire for other opinions.)

EDIT: I forgot to say that Feature is not yet merged into the main branch of Package, but it currently in a separate branch in the same git repository.

  • Could Feature be used on data not generated by Package? Oct 28 '19 at 20:14
  • @AzorAhai In theory. If the data used the same output formatting as Package (text files arranged in the file system in a certain way). Package is a modern rewrite of OldFortranCode, so output from OldFortranCode will also work. But it wouldn't be compatible with the main competitor to Package (unless someone writes the translation software). EDIT: Clarify: OldFortranCode was only used in-group; so it has no user base. Oct 28 '19 at 20:17
  • Would you expect people in other groups to ever use Feature to plot their data, then? Oct 28 '19 at 20:32
  • Feature is primarily designed for users of Package. Other groups will use Package, and therefore Feature (Feature will be distributed as a part of Package). In principle, someone could add a Competitor -> Package translator. But Competitor is a Python library, and therefore has immediate access to plotting with matplotlib, so I'm not sure its user base (of which I'm a member) will demand that. Oct 28 '19 at 20:38
  • I guess I don't really haven anything to add beyond Buffy's answer then. It does sound like you know the problem pretty well and this might be a publication-count stuffer Oct 28 '19 at 20:41

There is no bright line threshold for such things and the answer should be the same as for any scientific paper, hinging on questions of novelty and extension of knowledge. If it doesn't have that, then it probably isn't a good candidate, though the standards of different journals vary widely.

But you seem to describe a small-if-any advance with little novelty and you also seem to have made up your mind. We can't help you with the judgement and you just need to take a risk and call it. Others may disagree with you, but that is always the case in reviewing.

I don't really understand your point 5 and might disagree with your emphasis on it. Why is work done within a group less valuable that work that crosses institutional lines? Lots of papers are done by one or a few people within a research group. However, I might agree more if you mean all of their cited papers come from those same people. That isn't necessarily a red flag, but might be.

  • Sorry, to clarify point 5: I mean this in the sense that if Package had 100 authors, then the authors of Feature might feel like their contributions are getting lost in the mix, and I might understand why they felt a second paper was needed to emphasize their contributions. With ~6 authors, their contributions are not going to be ignored. Oct 28 '19 at 19:51
  • Also, edited to clarify that the journal explicitly forbids considering novelty. (Also impact, narrow focus, etc.) Oct 28 '19 at 19:59
  • Seems pretty broad. Is anything that is correct and doesn't break norms acceptable then?
    – Buffy
    Oct 28 '19 at 20:56
  • Yep. The journal intends to encourage publication of attempts at replication of prior experiments (assuming a justification can be given for the replication) and it allows negative/inconclusive results. So they don't want impact/novelty to be a measure for acceptance. I understand the intent for experiments, but I find this hard with software papers. So I'm trying to come up with a better test for "is this enough of a contribution for a publication" than "my gut says ...." Oct 28 '19 at 23:08
  • And before you ask (because it would be my next question) -- the journal is legit. Or was not long ago. I verified that one of the Nobel Laureates the parent journal lists still lists the parent journal as one of his editorial assignments. Oct 28 '19 at 23:11

I'm going to accept Buffy's answer, because even if it is not the "here's how to make it easy on you" answer I'd hoped for, it seems to be the generally-agreed correct answer.

That said, after a lot of thought (and considering feedback from here) I've come up with a list that pretty much satisfies me. I guess that everything after the first two points is more to check "is there novelty/notability here that you're overlooking because your gut reaction is that this isn't enough work to count as a paper?"

  • Does this implement any non-trivial science? If the tool implements significant data generation or data analysis logic, then it should be considered. "Significant" is still a subjective term, but just reading an output format would not be significant.
  • Is the software tool distributed independently? One important purpose of software papers is to make the community aware of plugins that otherwise wouldn't be noticed (lost in some dark corner of the internet). These may be relatively small (but useful) bits of code that would be overlooked without a journal to serve as an announcement forum. This criterion is probably only applicable to edge cases.
  • Is the tool re-usable in many contexts? Even a very simple tool can be a valuable contribution if it is designed to facilitate interoperability. For example, a simple "universal translator" for domain-specific formats might be very small, but might be worth publishing.
  • Is the approach novel? Even for journals that do not have a novelty requirement, the existence of a novel approach can be a consideration in whether a paper should be accepted.
  • Does implementation of the tool require significant domain expertise? Packaging significant domain expertise in a way that makes it available to non-experts is a potential purpose of a scientific software publication.

For the fifth point, I'm imagining something like a GUI that prevents users from selecting incompatible options, where the incompatibilities may require domain expertise. I know in my field we have script-based data generation programs where users can accidentally select incompatible options and get nonsense as a result.

I'm not positive that this covers all cases, but it covers everything I can think of. And I feel better having used a rubric that I could apply to any software package.

Posting with the hope of community feedback, and so the next person in a similar situation can see what I did!

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.