How to tell reviewers that I can't update my results

Question

I spent a lot of time during my post-doc doing a big-data analysis using cloud credits paid for by a grant. I got the research conducted, written up, and submitted to a journal. The modeling I did is not possible to conduct outside of a cloud environment.

Now, the paper has spent a long time in review, and that grant has expired, the cloud account has been closed, and all of the data has been deleted (although I still have the code).

The reviews are mostly positive, although some of them are requesting minor tweaks to the model, which is not possible. I also really don't think that the tweaks they are suggesting would improve the model.

What I am wondering is, how candid should I be about the grant situation? How much should I focus on the science versus my excuses for why I simply can't update the model?

Should I:

• Focus on the science and explain why I think the proposed changes to the models likely wouldn't affect the outcome?
• Discuss the grant details and tell the reviewers that our grant/cloud access expired, and therefore I can't re-run the models?
• Some combination, i.e., tell the reviewers that their proposed changes likely wouldn't affect the model findings, focusing on the science, but then tell the editor about the grant/cloud situation?

I'm also wondering what happens in other fields where experiments can only happen once, and it is impossible to update the findings.

Answer 1

A frequent response in experimental fields goes something like:

"We appreciate the reviewer's suggestion to use the 'X' technique, and do believe this could lead to an interesting extension of our current results. However, due to the unavailability of 'Y' resource, we are currently unable to undertake this study. We have, however, run a preliminary analysis* along the lines suggested, and found that it does not alter our main results significantly.

...some details/description of the analysis "

*The 'preliminary analysis' is often just a theoretical justification or back-of-envelope estimation. Its purpose is to establish that the suggested changes won't drastically affect your main claim(s). Including this in the response shows that you've genuinely tried to incorporate feedback within the available resources.

Be mindful of the conventions in your field though; @Kimball and @Anyon have pointed out that 'preliminary analysis' would convey something quite significant in their fields. If so, the response above may come across as flippant/disrespectful, and you may consider making a weaker claim, such as 'we estimate', 'we expect', 'our initial assessment is...'

Answer 2

As an addendum to the other answers, I would also suggest including a brief note about the total computing resources (CPU time, RAM, disk space) you've spent on modeling the problem in your paper (and possibly a more detailed description of the software and cloud computing environments used e.g. in an appendix). This has several potential advantages:

• It lets the reader (including reviewers) know that the modeling process was slow and costly, and not easily repeated with minor tweaks to the model, thus explaining why you have not explored such model variations further.

• It provides valuable information to anyone interested in reproducing or extending your results about the effort likely involved in doing so. In particular, if your reviewers underestimated the effort, it's likely that other readers may do so as well. Letting them know that your model was expensive to run may save them from wasting time and money on a project they cannot finish, which they may thank you for later.

• Conversely, it's possible that some other group might be able to carry out similar modeling much more easily, either because they have more money, faster computers (perhaps because they're reading your paper 10 or 20 years later) or simply more efficient modeling software. (Never discount the latter possibility, especially if you're using off-the-shelf software. It's amazing how often even minor algorithmic improvements can speed up a model by multiple orders of magnitude.) Knowing that the modeling task was challenging for you, with the software and computing resources you had, might inspire someone to write a followup paper showing how to model your system more efficiently (which, of course, would directly benefit your group too) or even to offer to collaborate with you on the problem.

In general, if an experiment you've performed was difficult to carry out for some reason, you should always make that clear in your paper — not merely to brag about doing it, or as an excuse for not doing it more times, but also as a challenge for others to find ways to do it more cheaply and efficiently. That is how methodology advances.

Answer 3

I would recommend stating the reasons why you don't think tweaking the model will make much difference but also mention that you no longer have access to the computing facilities necessary to re-run the model. You could also say that modifying the model could be part of future work (by yourself or others).

It's not unusual to be unable to repeat an experiment at a later date. I've done research with explosives. The test specimens, and sometimes the instrumentation, no longer exists after the test. Or at least is in very small pieces spread across a test site. Sometimes equipment is damaged and some data cannot be recovered. It costs a lot to do such tests and repeating them is simply not feasible financially.

Ultimately either the tweaks are essential and the work will be rejected, or the work is satisfactory as is and will be published (at this journal or another one).