4

Recently I have submitted my paper to IEEE Access Journal which has a binary peer review process, and I got rejected. Still, they allow me to answer the reviewers' concerns and resubmit again. One of the reviewers asked: "For the consideration of reproducibility, the code of the proposed method is suggested to be provided.

I want to know how I should respond to that concern politely as the author? I plan to provide all datasets and some part of the program, which is related to an example that I described in my paper. This simple example shows how my algorithm works and will help reproducibility (It shows the final result of applying the proposed method by considering one of the algorithms that I have used in my paper on a smaller dataset).

I want to know how to answer the reviewer politely and convincingly while explaining that I am providing some part of my programs, not all of it.

Thanks.

9
  • 8
    Do you have some reason for withholding part of it? – Buffy Dec 31 '20 at 16:00
  • 9
    Well, you can explain that to editors, but, since your paper was rejected and you need to re-write it, you may need to do more than you think. There are some battles you won't win. But, you haven't answered my question other than to say "time consuming". – Buffy Dec 31 '20 at 16:23
  • 2
    I would go ahead and release my code and results, even if it looks messy and needs work. – Coder Dec 31 '20 at 20:03
  • 8
    From your OP, the only reason you have given for not releasing it is "I don't want to". That is not going to satisfy anybody. But people here can't invent reasons and put them into your mouth. If the reviewer is concerned about reproducibility, that might imply he/she doesn't actually believe your results are correct. It wouldn't be the first (or last) time that software gave good-looking (but wrong) results because of a bug! We used to call the most insidious bugs the "20% errors". 2% errors usually don't matter. 200% errors are usually obvious. But 20% ... "here be dragons". – alephzero Jan 1 at 1:07
  • 1
    @alephzero But the best way to validate the results are to reimplement, not to pick over the existing code for bugs. A description of the method should be enough for this (along with the dataset). – Morgan Rodgers Jan 1 at 5:26
5

Let me try to capture the thoughts in the comments and add a bit, though I can't be specific, not knowing your work.

The reviewer has asked for your code. It might be appropriate or not to release it, depending on too many things to list here. But if you don't want to release it since future work depends on it and you prefer to keep it private for now, it might be an acceptable reason to an editor, or not. I can't predict that. If knowing it is essential to evaluating the correctness of your work, then the editor has a right to object.

But, the original was rejected and, if you want to resubmit to that journal, then you need to do a fairly extensive rewrite in any case. Evaluate for yourself how essential it is for you to not (yet) reveal things and if it isn't essential, try, within page limits etc, to comply with the request. It might make the paper better and it might send it off into inessential areas resulting in cruft. But the decision is yours to make how to write up your work. And the editor's job is to judge its acceptance for the journal.

But the bottom line is that you don't need to reply to every concern or accede to every demand. The work is your own. But your readers, including the reviewers and editors will make judgments about it as to whether you make the case for any conclusions you hope to draw. Make sure that the proper connections are made.


I have a guess that you are worried about getting scooped on future work if you reveal too much. That may be a valid concern. If so, releasing a more complete version of the current paper close in time to the follow up might be a solution, so that there is less opportunity for others to jump on your methodology and break in line ahead of you.

16

You cannot answer “convincingly”, because your assumption that it is okay not to provide the code is incorrect, and the reviewer is correct to ask for it. And the fact that it’s time-consuming to prepare the code for release is neither here nor there: it’s also time-consuming to write a good paper and polish it over and over to make it into something people can read and understand; but that’s what good researchers do, because they understand it’s important and that this is the way to do good science.

For a related discussion, see here.

3
  • 1
    I very much agree with your linked answer, that researchers should release their code by default (with rare exceptions), and that the main counterarguments boil down to laziness at best. But what you write here is a little misleading — there are many fields in which, according to current norms, it’s perfectly normal and acceptable to not release code. I greatly hope those norms will change soon, but for now, they are what they are, and it’s a bit misleading to pretend otherwise. – PLL Jan 1 at 14:45
  • @PLL thanks, I’m not pretending anything, it’s just my opinion and others are free to disagree. However since OP’s paper got rejected it seems that the referee shares my view and is similarly unlikely to find OP’s arguments for not sharing their code convincing. – Dan Romik Jan 1 at 18:25
  • Fair enough, and I greatly agree with your opinion and agree it sounds like the referee shares this position. It was just that (to me) the first sentence of your answer reads like it’s presenting an accepted consensus, rather than a (common but far from universal) opinion. – PLL Jan 1 at 22:57
7

There are no models for this. I think you have answered your own question here:

answer the reviewer politely and convincingly while explaining that I am providing some part of my programs, not all of it.

You suggest several reasons why not all the code is called for. Some is extraneous. Some is working but not fit for public consumption (all too often true). Some you may want to keep private for future use. Those reasons might satisfy the editor.

What you do provide might be in a web accessible repository - not necessarily written up as part of the paper. It's becoming more common for journals (and editors and reviewers) to require at least that much public exposure of data and code.

1
  • 3
    I would point out that the reviewer did not specifically ask for all the code to be made public. It certainly is reasonable to release some code and not other code, for all manner of reasons. – Terry Loring Dec 31 '20 at 20:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.