17

I'm reviewing a manuscript that is protocol focussed. This protocol is a bioinformatics workflow that uses the author's own software that has already been published.

Part of the peer review requirements are to test the protocol to see if the instructions are accurate. I have no problems with this request. However, although their website suggests that the software should be installable on any OS, the installation package only includes an .exe file for a Windows system. I do not have a Windows system.

It is possible for me to figure out how to install their software, but these instructions are not included in the paper nor on the software website. This would also take me some time and this would be more than a normal amount of time that I should be spending reviewing this paper.

My question is: should I only review what I am able to easily complete from their protocol? In other words, should I only review what I can do from their given instructions and include the difficulties I had installing the software? Or should I contact the authors (without mentioning I am reviewing their new manuscript) and let them know that their installation package for their already published software is missing important files for unix-based systems?

I am still relatively new to the peer review process and think this paper is very interesting and don't want it to be rejected simply because I do not use Windows.

| improve this question | |
  • 10
    If the authors of the paper only use Windows, it seems totally unreasonable to expect them to port their software to a different OS and then re-validate it on that OS solely for the purpose of a review. And even if they did that, you are not actually reviewing what they did. There is a huge difference between "it should be installable (in principle) on a different OS" and "we claim that it works correctly on a different OS". – alephzero Dec 21 '19 at 1:50
  • 20
    Is it really that hard for you to temporarily get access to a Windows machine and run their exe? For instance, you could ask a colleague across the hall if you can borrow their computer, or you could spin up a vm (they are free). – Nate Eldredge Dec 21 '19 at 6:35
  • 12
    if you're using a reasonably modern system, you could just install a Windows virtual machine using any of VMWare/VirtualBox/Qemu and run the exe in the Windows VM – hanshenrik Dec 21 '19 at 8:21
  • 10
    Beside of a VM, wine (winehq.org, applied e.g. for Ubuntu vitux.com/how-to-install-wine-on-ubuntu) and PlayOnLinux (en.wikipedia.org/wiki/PlayOnLinux) or CrossOver for MacIntosh (en.wikipedia.org/wiki/CrossOver_(software)) equally allow to work with some orginally-Windows programs in your host OS, too. Not all do fantastic for each application, of course, but for this you the different compatibility levels, e.g. silver/gold/platinum in wine. – Buttonwood Dec 21 '19 at 17:12
  • 7
    Claiming that it is not dependent on Windows and then acting like no one uses anything else should be mentioned in the review. And depending on the nature of the software, it may be wrong to assume most readers can do the VM tricks in these comments. – WGroleau Dec 21 '19 at 17:55
34

You should ask your editor for advice on this. I doubt, however, that you should reach out to the authors in any way. But the editor can, on your behalf, ask the authors for specific instructions for installing the software on your OS. The editor can make this a condition of getting a review.

Moreover, the editor can communicate your concerns about missing files or whatever else is important to complete a review.

| improve this answer | |
  • 3
    Yes, you should not lightly break the anonymity of the review process. That could lead to trouble down the road, for example if you find that you have to give a negative review for any other reason. Going through the editor is much better. – ObscureOwl Dec 20 '19 at 19:35
  • 1
    This might be too much work for the editor and for the OP. – user1271772 Dec 21 '19 at 21:13
  • 1
    This is exactly what I decided to do. I feel like the file omission was an oversight and I'd like to give the authors a chance to remedy it before I submit the actual review. – tinyteeth Dec 22 '19 at 19:18
  • 1
    Come on. As much as I love Linux and as much as I like macOS, isn't it a bit ridiculous to contact the editor and authors asking for help because they "only" provided an exe for Windows? – Eric Duminil Dec 23 '19 at 15:08
  • Following up on this, I ended up contacting the editor who contacted the authors. They had accidentally left out essential files for installation for any operating system. BTW @EricDuminil, it was not "a bit ridiculous" because the review could not have been completed without it :) – tinyteeth Mar 10 at 20:44
6

If the protocols/software implementation are a requirement for the journal rather than just the concepts, which sounds like the case, I would say there are two scenarios:

1.) Given a reasonable effort and appropriate background on your part, you were not able to get their software to run. All else in the paper seems fine. In this case, I might suggest "Reject and resubmit". Given that you are not able to test their code, you can't really give it a thumbs up or thumbs down, but it should be a relatively small step for them to meet this requirement and then you can adequately evaluate their software.

2.) You are able to use their software, but it took an unreasonable amount of effort or expertise on your part to get it done. In this case, you can evaluate their software, but ideally they should make it more readily available to the target audience. In this case, under the condition that you liked everything else about the paper, it may be reasonable to suggest "Accept with minor/major revisions", and you ask them to include the necessary instructions that will help guide other users of the code.

| improve this answer | |
  • Comparing to replication in other sciences, which would require looking at the exact same live organisms, or unusual chemicals, or expensive specific hardware for the experimental setup, I'd assume that getting access to a computer with a somewhat popular OS version falls under "reasonable effort". If the software runs only on a weird, obsolete system that's not widely available, then running it would require unreasonable effort, but getting access to one of the 80%+ worldwide computers that run Windows definitely is not. I mean, Linux-only (2% marketshare) software gets reviewed as well. – Peteris Dec 21 '19 at 11:23
  • 1
    @Peteris: I guess I didn't say it explicitly; I believe the OP's scenario is (2), so conditional on them being okay with the rest of the paper, I'd say "Accept with revisions" and ask them to give installation instructions for linux/unix as that doesn't seem like a huge request for the authors. – Cliff AB Dec 21 '19 at 15:14
  • @Peteris let's not conflate marketshare for Facebook batteries with marketshare of people who do real work on their computers. That being said, yeah, OP should beg-borrow-steal a Windows PC for the test run. – Jared Smith Dec 23 '19 at 17:19
5

It sounds from your description that the software is an essential part of the paper. Well, if the paper was missing one of its normal text sections due to sloppiness of the authors, would you be blaming yourself for that and going on a wild goose chase around the internet to find the missing section? I suspect not. The same should apply here. As referee, it is not your job to become an expert in compiling Linux binaries from source code (or whatever similar tasks you apparently need to do to get a working binary to test).

Similar to what @Buffy suggested, what I would do is write to the editor and inform them that I am unable to finish the refereeing assignment since not all parts of the work that need evaluation were provided to me. I would make clear that this is not a rejection, and that I am happy to finish the job once this situation has been remedied. It’s the editor’s job to communicate with the authors, so leave that part to them.

| improve this answer | |
  • 10
    Saying the software cannot be run under linux is like the paper missing a part seems a bit over the top: Did the editor specify that the software should run on OP's OS? To me it reads more like OP speculates from the webpage that it should be installable on any OS, perhaps that's more a goal description and not the current state relevant for the paper? I do agree that the instructions should be clear so OP knows whether he can expect it to run on Linux or whether he would need a windows machine to run it. So, improvement suggestion: focus on the missing doc, not the missing OS support. – Frank Hopkins Dec 21 '19 at 4:40
  • 3
    @FrankHopkins agreed. In fact, it certainly ought to be acceptable (and is acceptable AFAIK) for software that’s supplementary to a submitted paper to run on only one of the major OS platforms (assuming other conditions are met, particulary if the source code is available). However, that doesn’t mean the referee should have to tolerate misleading claims by the authors about which platforms are supported. And in any case, if OP does not have access to Windows computers then practically speaking they cannot referee a paper whose software component is Windows-only, and should let the editor know. – Dan Romik Dec 21 '19 at 5:32
  • @DanRomik Which "misleading claims" are you referring to? I see no evidence that the author claimed the program would work on any platform besides Windows. – Lightness Races in Orbit Dec 22 '19 at 17:43
  • @LightnessRaceswithMonica The authors did not specify on which OS the software could run, but their primary manuscript and software website say it has been tested on both Windows and Linux and that their installation package should be including both an .exe and .sh file. However, the .sh file and separate .jar file is missing. – tinyteeth Dec 22 '19 at 19:20
  • @tinyteeth Okay. It's just that, unless I'm missing something, you hadn't stated that when Dan made the claim. – Lightness Races in Orbit Dec 23 '19 at 2:39
2

In bioinformatics, it is quite common to have to install software from source., however they should also have good instructions such as vsearch.

Although vsearch also does have premade binaries, if this is early on in the softwares development and they are happy with how the algorithm works I don't think it is unfair to expect users to compile from source (again given good instructions).

If you actually can't install it to evaluate the content of their paper either ask for help from someone who might be able to or ask for good install instructions, if the software has not been released or is in beta then they might not be expecting a lot of people to install it yet. If you can install the software to evaluate the paper I would say yes but... and express your concerns about how the software isn't very beginner friendly.

| improve this answer | |
2

One thing I haven't seen in any of the other answers so far, is to politely tell the editor that you cannot review this paper. The editor will send it to a different referee, who likely will be able to get access to a computer that had Windows.

I'm reviewing a manuscript that is protocol focussed...
Part of the peer review requirements are to test the protocol to see if the instructions are accurate.

You have the option to test the protocol and review the paper (which seems like it will require a lot of effort), or you can choose not to review the paper at all (which will be very easy for you to do, in terms of the workload).

It all depends on how much effort you want to put into it, and whether or not you think that effort is worthwhile (for example building a good relationship with the editor by generously donating your time towards helping their journal, or having the chance to have your input considered before this paper is published). Sadly this referee task seems to be more time-consuming than the typical one, but again: no one is forcing you to referee this paper.

| improve this answer | |
1

You did not make clear whether you made any attempt to install software on a Windows machine. If you did, then:

First:

Part of the peer review requirements are to test the protocol to see if the instructions are accurate.

Then:

It is possible for me to figure out how to install their software, but these instructions are not included in the paper nor on the software website.

Isn't this obvious? Instructions are incomplete/inaccurate/unavailable and the paper should be assigned a major review round based on that fact alone.

If you don't and cannot get access to windows computer, then you cannot do the review, I am afraid, and you should let the editor know (without any penalty to authors).

Never contact authors directly!

| improve this answer | |
1

.exe does not always mean a windows executable. .NET applications have the .exe extention by default, so your application may be executable with mono name.exe.

If it really is a Windows application, that's likely also fine. wine, installable on MacOS, Linux and FreeBSD is able to run most Windows applications just fine.

| improve this answer | |
  • I guess this isn't necessarily the problem. The manuscript is basically a tutorial on how to use the software, and is not the primary manuscript for the tool. The tutorial should have these instructions. – tinyteeth Dec 22 '19 at 19:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.