56

I have just received a decision letter for my submitted manuscript to an Elsevier journal. It was a revise and resubmit. However one of the reviewer asked for an executable file in order to check my results. (I felt distrust from his comment..)

This is regarding a computer science paper on testing the efficiency of an algorithm on a set of instances from the literature. I compared the results of the algorithm with those of other authors.

  • 31
    To clarify: the paper talks about some software, and the reviewer wants to be able to run the software? I don't know how common that is (I'm not a computer scientist), but so long as any license on the software permits you to give it to the reviewer it does sound like a reasonable request. – Flyto Apr 27 '17 at 21:40
  • 18
    @DSVA: "If you publish an algorithm you need to somehow publish the actual code"... what percentage of published algorithms do you think come with source code? – Mehrdad Apr 28 '17 at 5:47
  • 87
    @Mehrdad "what percentage of published algorithms do you think come with source code" a lot less than the percentage that should come with source code. Imho if you can't verify the claim without implementing the algorithm, then it should have source code attached, and the source code should be reviewable, or else it's not good science. No reviewer in their right mind would accept a paper with "magic happens here" in the middle of the proof so "closed source software happens here" shouldn't be allowed either. – Sumyrda Apr 28 '17 at 6:42
  • 40
    A distrustful reviewer is a good thing, it means they are going to give your paper a stern test, and if there are problems with the paper, they are more likely to find them, which is to your long term benefit. I very much doubt they distrust your honesty, just the result. – Dikran Marsupial Apr 28 '17 at 9:25
  • 6
    What other way do you suggest should he go to verify your results? – PlasmaHH Apr 28 '17 at 14:29
171

I don't know if it's normal, but it should be normal for all reviewers to make reasonable efforts to verify that the claims authors make are correct, so to the extent that it's not normal, I can only commend the reviewer for being willing to make an effort that other reviewers don't make. What you sense as "distrust" is the reviewer doing their job, nothing more or less (and it is probably somewhat accurate to say that a reviewer's job is to distrust the author's claims, so I don't see the idea of being distrusted by a reviewer as something to be ashamed of or offended by).

By the way, it should also be normal for authors to make available any software (including source code whenever possible) needed to replicate and verify their results. So if you are unhappy with the reviewer coming back to you with annoying requests that delay the decision on your paper, next time around you can preempt such issues by releasing your source code (or at least submitting it to the journal) alongside your manuscript. I am sure the reviewer would be much happier and ultimately everyone would benefit, including you.

  • 4
    Thank you for your comment, perhaps I felt offensive because it is my first journal paper. However, in my research field it is not common that authors make available their code.. Anyway, i will send the executable to the reviewer because he stated that without this check the paper cannot be accepted . Thank you again. – Marvel LePont Apr 27 '17 at 23:12
  • 92
    As someone who works in the industry and reads papers as "an outsider", it has always struck me as very odd that there's a tendency to "hide the code". My feeling is that code, just like a paper, should be out there and available. I'm absolutely with Dan here, it should be done more. To me at least, not publishing code always has a bitter aftertaste of there being something dodgy going on. – SBI Apr 28 '17 at 7:38
  • 12
    @SBI It's not odd at all. If you invest 4 years of work into a code, you need it to pay off in enough papers/citations/attention to get you a promotion. One paper will not do that, and if you release your code, you'll need to spend another lengthy period of time coding up a new publishable result while your rivals use your code to scoop you. It's not so much "publish or perish" as "publish more than alternative hires or perish". (Personally, I did release my code; I also perished.) – Xerxes Apr 28 '17 at 13:41
  • 4
    Another red flag that catches my attention is that they request an "executable file". The user has no way to verify that the executable doesn't just print the results with no algorithm behind, aside from reverse-engineering it (without going into too much detail here). Such request can possibly mean that the reviewer tries to lull the author into a false sense of security by intentionally not requesting the source code. – user3209815 Apr 28 '17 at 14:53
  • 20
    @Xerxes How can your idea be publishable in a paper though, if it depends so much on closed source code? I mean, you lay open your entire idea anyway. I've written code basing on papers countless times. Usually, code is more a proof of concept rather than holding more information than what you publish anyway. The other way around though, it becomes trickier. Making claims without showing they actually work is... difficult. – SBI Apr 28 '17 at 15:32
61

I come from a different field, in which the code we use isn't a major output. But if a referee asked for the code, we would provide it and happily. Most of our work is done in python so an executable wouldn't be usual, the source would (also true for matlab).

In fact the only thing I find slightly odd here is the use of executable rather than source.

Don't be offended by the request for a couple of reasons: It's not the reviewers' job to trust you; it's their job to check your paper. If a reviewer takes enough interest in your work to want to run your code, they haven't dismissed your paper out of hand.

  • 13
    I like the final point. It would even seem a bit honouring to me if the referee would be willing to try out my software :) – Džuris Apr 28 '17 at 8:32
  • 4
    @Džuris indeed, it means they are taking your paper very seriously. – Dikran Marsupial Apr 28 '17 at 9:42
  • 7
    Thank you for your second paragraph! No one here seems to see this. When I read the title of the question, I assumed OP found it odd that someone would request an executable rather than the source code because that indicates they don't care enough to check the source code for errors / mistakes / deliberate mistakes and in fact are too stupid or lazy to compile the source code (which I assumed was provided) themself. – UTF-8 Apr 28 '17 at 12:59
  • 2
    @UTF-8 It's also possible they don't have a compiler (maybe can't because of licensing/platform issues) or don't have the skill in that language to make sense of the source. Even then I'd still want the code – Chris H Apr 28 '17 at 13:05
  • 4
    Upvoted and a ++ for the fact that it is weird they want an executable and not the source. Heck when I grade homework, I want the source and anything else required to get it to compile, including compiler commands/statements/options/arguments if it is anything more complex than "g++ foo.cpp -o foo" – ivanivan Apr 28 '17 at 21:36
14

To summarise the situation with your data:-

1) You came up with an algorithm on paper/Matlab/whatever.

2) You implemented that algorithm in some programming language.

3) You built a set of test data to exercise your algorithm, and came up with some results for what it should do in theory.

4) You put that test data through the code and came out with some results for what it does in practise.

In this process there are various places where things can go wrong with your methodology. Your code may not correctly reflect your algorithm. Your test data may have been worked backwards from the code instead of forwards from the algorithm. Your test data for your algorithm and your test data for your code may not be the same.

Unless the reviewer has the algorithm and the source code and all the test data for both and all the output data for both, they cannot verify that your work is sound and your conclusions are valid. This is not subject to dispute - it is logically impossible, if they want to properly review your work. Anything else is making assumptions which may not be valid.

I have personally been affected by this situation, when my company bought some control theory IP from a researcher. He'd written papers on how this was supposed to work and the theory behind it, and then he'd built some electronics to implement his theory. His papers covered the theory, and also included schematics for the electronics. When I read this to work out how to implement his theory in software, I found that the schematic had an extra filter in it. The action of this filter turned out to be critical to the system being stable or even effective, but it was not documented at any point anywhere in his work. It wasn't until we had a phone call with him that we found out what the purpose of the filter was, and how we were supposed to tune it.

This was in a paper which theoretically had been peer reviewed when it was published. Clearly it hadn't been peer reviewed thoroughly enough! His results showed that given the same data, the implementation output was pretty close to the theoretical expected output, and the effect of the filter was at a different place in the response. Still though, the implementation flatly would never have worked without this filter present, and it wouldn't have been at all hard to include this in the theoretical model. He could even have said "this filter is required for these reasons, but can be ignored in this area of the response we're looking at for these reasons" and he would have been covered. What is not acceptable is what he did, which is to fail to mention it at all, because the end result of that is that someone trying to implement his work would be unable to.

Like I said, he still got his paper published, and no-one complained at the time. It should have been spotted by his original reviewers though. In your case, your reviewer should be looking for discrepancies like this - it's the whole point of peer review. So if people are asking you for things you haven't made available, (a) it's a good sign they're checking thoroughly, and (b) you should have made it available in the first place as best practise.

6

Artifact submissions are a thing in CS. What I've seen is that you'd prepare a virtual machine, where your software is already set up and ready for making experiments. So, the reviewer may be referring to that the journal has some official procedure for artifact submissions. Alternatively, some authors just make the source code of their tools and benchmarks available via services like github, and the reviewer may be suggesting you should also do this. Regarding the distrust, computing people are naturally wary about benchmarks and tool comparisons, as the final figures may depend a lot on how your experiment is set up (e.g., if you compare to your own implementation of an existing algorithm, did you implement it correctly). It could also be that the numbers that you give in the paper seem a bit odd, but then the reviewer would have pointed to what exactly doesn't look right to them.

3

Submitting an executable isn't the same as submitting source code. An executable doesn't really give the recipient any access to your original code (as a computer science student should already know, of course). I don't see a problem with this request.

  • 5
    I see a major problem: the executable run on the same test cases is going to produce the same results. even if those results are wrong due to an error in the program. – jamesqf Apr 28 '17 at 5:20
  • 7
    I'm curious how you would send an "executable" of, say, Python code, without giving access to your original code? Are you expected to obfuscate it? – Mehrdad Apr 28 '17 at 5:48
  • 4
    @jamesqf They could try other cases, not covered by the authors. – Captain Emacs Apr 28 '17 at 6:06
  • 1
    Any executable can be decompiled to produce functionally the same code. Any code that compiles to JVM or .NET can be decompiled to something relatively close to the original, and even machine code can be torn apart with enough work. While I deeply believe source code should be published with publications like this, if you refuse to do so, you can't offer an executable as if it will hide what you wanted to hide from your source code. – prosfilaes Apr 30 '17 at 1:06
  • @CaptainEmacs but they can't do that if they can't see what the program actually does. It might as well have data embedded in the executable or do something else than what is claimed. – mathreadler Apr 30 '17 at 15:41
2

Given my personal experience with open source communities and the assumption that the paper includes the entirety of the algorithm in question, then sending the source code or related compilation of said software wouldn't produce many negative effects.

This would allow the reviewer to verify results and claims made by the paper's author. The key issue the reviewer might be looking for is that you correctly implemented algorithm in source code an are not mistakenly relying on a feature of the programming language, OS, or hardware to make claims about its running time or other features.

Off the top of my head I would relate that in I/O bound cases its easy to mistake efficient algorithms for, as an example, Javascript's ability to make almost every function call asynchronous. Of course this is mostly seen in I/O bound operations rather than proliferative computational loops. Then the efficiency measured is not that of the algorithm as a formal proof but; instead it relies on a language specific feature.

The salient point is that there are many cases in which the formal algorithm and the implementation can diverge from representing each other faithfully and in doing so the conclusion, if based on empirical metrics such as running time, can run into many issues where an improper implementation can attest to an incorrect conclusion.

1

Source code can have bugs, and to truly effectively review an algorithm, a prose description of the method alone may well be insufficient. Sharing something beyond the text is beneficial; a good paper with the actual source code (+sample inputs) is the gold standard for reproducibility.

One fun wrinkle: depending on where your reviewer is, you might not be allowed to give them a binary. Eg, some code uses proprietary libraries that are licensed freely in academia, but someone in industry might require a separate license to even use an existing binary, much less compile it. (this happened to me once, though not as part of peer review)

0

It is an idiotic request on his part.

  1. He could catch a virus.
  2. There is no realistic way he can check that the executable implements what is described in your paper, ergo no way the request has any scientific value whatsoever.

He should be asking for the source code, and that is all you should agree to give him.

  • 9
    #2 is often false. A significant number of problems have the characteristic that checking a solution is much easier than finding a solution. In such a case, a reviewer can verify that a black box does indeed produce correct solutions and measure the runtime complexity. This would be particularly valuable if, for example, the reviewer noticed that all the examples used in the paper had particular characteristics that made them easier to solve than the general case, as he could formulate his own test cases. – Ben Voigt Apr 29 '17 at 3:52
  • 1
    @BenVoigt But what black box? How can the reviewer know he has the black box implementing what the paper claims? – user207421 Apr 29 '17 at 6:03
  • 1
    There is a valid point that use of a black box doesn't assure that paper contains an accurate description and explanation of the method used -- but the reviewer may be much less worried about the risk of having faked the description given that a novel method is proven to exist, at least compared to the risk of faking both method and description. – Ben Voigt Apr 29 '17 at 6:08
  • 4
    The reviewer could run the executable with a different set of parameters to reproduce a known result. OP doesn't seem to give us enough information to know that it is the case. As for the part with the virus, a Linux user might feel safer than a Win user. – user21264 Apr 29 '17 at 12:03
  • 3
    @Magicsowon “a Linux user might feel safer than a Win user” Linux user here. I don't agree at all. You could easily write a program that damages personal documents or sends them through a network connection on both platforms, if the recipient is kind enough to run it for you. Linux is safer for "widespread" malware that is not targeting you specifically, also because there are more which are made for Windows. – Andrea Lazzarotto Apr 30 '17 at 17:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.