I have just received a decision letter for my submitted manuscript to an Elsevier journal. It was a revise and resubmit. However one of the reviewer asked for an executable file in order to check my results. (I felt distrust from his comment..)
This is regarding a computer science paper on testing the efficiency of an algorithm on a set of instances from the literature. I compared the results of the algorithm with those of other authors.
I don't know if it's normal, but it should be normal for all reviewers to make reasonable efforts to verify that the claims authors make are correct, so to the extent that it's not normal, I can only commend the reviewer for being willing to make an effort that other reviewers don't make. What you sense as "distrust" is the reviewer doing their job, nothing more or less (and it is probably somewhat accurate to say that a reviewer's job is to distrust the author's claims, so I don't see the idea of being distrusted by a reviewer as something to be ashamed of or offended by).
By the way, it should also be normal for authors to make available any software (including source code whenever possible) needed to replicate and verify their results. So if you are unhappy with the reviewer coming back to you with annoying requests that delay the decision on your paper, next time around you can preempt such issues by releasing your source code (or at least submitting it to the journal) alongside your manuscript. I am sure the reviewer would be much happier and ultimately everyone would benefit, including you.
I come from a different field, in which the code we use isn't a major output. But if a referee asked for the code, we would provide it and happily. Most of our work is done in python so an executable wouldn't be usual, the source would (also true for matlab).
In fact the only thing I find slightly odd here is the use of executable rather than source.
Don't be offended by the request for a couple of reasons: It's not the reviewers' job to trust you; it's their job to check your paper. If a reviewer takes enough interest in your work to want to run your code, they haven't dismissed your paper out of hand.
To summarise the situation with your data:-
1) You came up with an algorithm on paper/Matlab/whatever.
2) You implemented that algorithm in some programming language.
3) You built a set of test data to exercise your algorithm, and came up with some results for what it should do in theory.
4) You put that test data through the code and came out with some results for what it does in practise.
In this process there are various places where things can go wrong with your methodology. Your code may not correctly reflect your algorithm. Your test data may have been worked backwards from the code instead of forwards from the algorithm. Your test data for your algorithm and your test data for your code may not be the same.
Unless the reviewer has the algorithm and the source code and all the test data for both and all the output data for both, they cannot verify that your work is sound and your conclusions are valid. This is not subject to dispute - it is logically impossible, if they want to properly review your work. Anything else is making assumptions which may not be valid.
I have personally been affected by this situation, when my company bought some control theory IP from a researcher. He'd written papers on how this was supposed to work and the theory behind it, and then he'd built some electronics to implement his theory. His papers covered the theory, and also included schematics for the electronics. When I read this to work out how to implement his theory in software, I found that the schematic had an extra filter in it. The action of this filter turned out to be critical to the system being stable or even effective, but it was not documented at any point anywhere in his work. It wasn't until we had a phone call with him that we found out what the purpose of the filter was, and how we were supposed to tune it.
This was in a paper which theoretically had been peer reviewed when it was published. Clearly it hadn't been peer reviewed thoroughly enough! His results showed that given the same data, the implementation output was pretty close to the theoretical expected output, and the effect of the filter was at a different place in the response. Still though, the implementation flatly would never have worked without this filter present, and it wouldn't have been at all hard to include this in the theoretical model. He could even have said "this filter is required for these reasons, but can be ignored in this area of the response we're looking at for these reasons" and he would have been covered. What is not acceptable is what he did, which is to fail to mention it at all, because the end result of that is that someone trying to implement his work would be unable to.
Like I said, he still got his paper published, and no-one complained at the time. It should have been spotted by his original reviewers though. In your case, your reviewer should be looking for discrepancies like this - it's the whole point of peer review. So if people are asking you for things you haven't made available, (a) it's a good sign they're checking thoroughly, and (b) you should have made it available in the first place as best practise.
Artifact submissions are a thing in CS. What I've seen is that you'd prepare a virtual machine, where your software is already set up and ready for making experiments. So, the reviewer may be referring to that the journal has some official procedure for artifact submissions. Alternatively, some authors just make the source code of their tools and benchmarks available via services like github, and the reviewer may be suggesting you should also do this. Regarding the distrust, computing people are naturally wary about benchmarks and tool comparisons, as the final figures may depend a lot on how your experiment is set up (e.g., if you compare to your own implementation of an existing algorithm, did you implement it correctly). It could also be that the numbers that you give in the paper seem a bit odd, but then the reviewer would have pointed to what exactly doesn't look right to them.
Submitting an executable isn't the same as submitting source code. An executable doesn't really give the recipient any access to your original code (as a computer science student should already know, of course). I don't see a problem with this request.
Given my personal experience with open source communities and the assumption that the paper includes the entirety of the algorithm in question, then sending the source code or related compilation of said software wouldn't produce many negative effects.
This would allow the reviewer to verify results and claims made by the paper's author. The key issue the reviewer might be looking for is that you correctly implemented algorithm in source code an are not mistakenly relying on a feature of the programming language, OS, or hardware to make claims about its running time or other features.
Off the top of my head I would relate that in I/O bound cases its easy to mistake efficient algorithms for, as an example, Javascript's ability to make almost every function call asynchronous. Of course this is mostly seen in I/O bound operations rather than proliferative computational loops. Then the efficiency measured is not that of the algorithm as a formal proof but; instead it relies on a language specific feature.
The salient point is that there are many cases in which the formal algorithm and the implementation can diverge from representing each other faithfully and in doing so the conclusion, if based on empirical metrics such as running time, can run into many issues where an improper implementation can attest to an incorrect conclusion.
Source code can have bugs, and to truly effectively review an algorithm, a prose description of the method alone may well be insufficient. Sharing something beyond the text is beneficial; a good paper with the actual source code (+sample inputs) is the gold standard for reproducibility.
One fun wrinkle: depending on where your reviewer is, you might not be allowed to give them a binary. Eg, some code uses proprietary libraries that are licensed freely in academia, but someone in industry might require a separate license to even use an existing binary, much less compile it. (this happened to me once, though not as part of peer review)
It is an idiotic request on his part.
He should be asking for the source code, and that is all you should agree to give him.
