1

I'd like to get the software-sharing aspect of my research paper right. (This empirical paper in health economics relies on code for modelling and simulation.) Besides open-sourcing the code on GitHub, what other aspects could be valuable to readers? For example:

  • Well-structured (clean) code
  • Tests
  • Documentation
  • Usage examples
  • GUI

Knowing this would help me prioritise the development efforts.

9
  • "Would it be to check the code for bugs, validate the paper methodology, reproduce the results or apply it to similar problems?" - Yes.
    – Bryan Krause
    Commented Apr 17 at 15:56
  • @BryanKrause You'd do all of these? But which aspects should I prioritise? Commented Apr 17 at 15:58
  • 1
    What would it mean to prioritize those things differently? How are you going to provide software that validates the methodology and reproduces the results but cannot be used to check for bugs? Are you creating something new that would be reasonable to apply to similar problems or are you using existing packages where someone would be better off using those packages and documentation directly?
    – Bryan Krause
    Commented Apr 17 at 16:00
  • @BryanKrause For example, I'd decide how much to focus on adding unit tests. Normally, do you comb through thousands of lines of spaghetti code submitted with a paper? Commented Apr 17 at 16:09
  • 2
    Overview, examples, installation instructions, clear versioning, paper citation info. The code base will change. Commented Apr 18 at 2:14

2 Answers 2

3

Congratulations on open sourcing your code! If only more researchers shared your interest in software quality :)

As to what to do next:

  • Well-structured (clean) code
  • Tests
  • Documentation
  • Usage examples
  • GUI

None of the above! (Well, not exactly.)

The most effective thing you can do to increase the likelihood that other researchers will be able to reuse your software is probably:

  1. Provide clear and detailed installation instructions, including what operating system(s) the code will run on, and what dependencies are required; and

  2. Try out the installation instructions yourself, on a fresh machine with no other software installed, to see if it works.

The idea here is that if other researchers can't even run your code, it is unlikely that tests, clean code, or documentation will help. You might be surprised how often this happens! Sadly, in my field (though I don't know about yours), perhaps 9 out of 10 software projects fail to replicate at this stage, due to various missing dependencies, broken build scripts, or platform incompatibilities. The problem only becomes worse with time.

Depending on your field, other solutions may be worth investing in:

  • For example, creating a website where your tool can be accessed / tried out, if feasible, can be a good way to increase accessibility, but takes a greater time investment and a web hosting service.

  • Providing a pre-built binary that will work on common machines (e.g., x86 windows and Mac OS M3) could also be worth considering.

Finally, if you are worried about long-term availability of your work (for instance, after current operating systems and dependencies become obsolete), you can also package up a version of your code with a virtual machine software like VirtualBox. This is a common solution for long-term digital artifacts, used for example by the ACM Digital Library. The downside of this is that downloading and configuring VirtualBox often makes it difficult to reuse your work in the shorter term, at least, for all but the most determined users. So it should not be considered a replacement for good installation instructions.

2
  • The installation/deployment part is something I did not prioritise -- that's very helpful to know! I guess Docker is a no-no unless it's for CS people. Would running pre-built binaries get scary messages from Mac OS, like "danger: unknown developer", and frighten some people off? Commented Apr 18 at 15:41
  • 1
    @schrödingcöder It turns out that Docker isn't even cross-platform. It's a common misconception, as it used to be largely true (through emulation), but is no longer true for newer macs (M1-M3). About the unknown developer warning, I wouldn't be too worried, but that's another thing to mention in your installation instructions :) Commented Apr 18 at 18:51
-1

There is a big misunderstanding, and it starts with your besides.

Besides open-sourcing the code on GitHub

Using a centralized service as Github as the repository of one's own code is ... damaging open-source. And science in the long-term.

An example of some critiques can be found here: https://sfconservancy.org/blog/2022/jun/30/give-up-github-launch/

The rest you propose is all good, in principle, but in the end a software is as good as the assumptions behind it and as bad as the bugs it includes, so spending time on these two extremes (assumptions = scientific publications, bugs= bugs hunting) is all you should do, and the rest is "nice to-have" (if your days are 36 hours long ;D ). Try to document the code on the go (Doxygen and similar systems), so you do not need to do afterwards.

Good luck!

4
  • I didn't see GitHub this way, but the link does have valid points -- they're good to know. But GitHub is so enmeshed in my and other's workflows, a bit like an addiction. I wonder if you've looked into alternatives and could recommend something decent. Commented Apr 18 at 15:48
  • 1
    As an alternative to Github, I once worked with a compnay having a private gitlab server. It was working smoothly, but I am not sure if the same critiques apply to gitlab as well. For simple access to code, there is Zenodo about.zenodo.org which is open, public-funded (i.e. not for profit) and in principle there are clear clause whether an AI can scan its content and use for its own purposes.
    – EarlGrey
    Commented Apr 18 at 16:32
  • 1
    While I agree with the criticisms of GitHub, I don't see how simply putting your code there is "damaging open-source." In fact if you attach a permissive license (e.g., MIT or CC BY) anyone can copy your code and later upload it to some other platform anyway. Commented Apr 18 at 18:57
  • @CalebStanford exactly!
    – EarlGrey
    Commented Apr 19 at 6:07

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .