How relevant is "moving on" as an argument in peer review?

Question

For me, publishing results only when they are final, complete, and positive is more of a tradition than a necessity. I would find it valuable if others were able to publish what they have tried, even if they have not achieved what they (initially) aimed for. Unfortunately, this is very often not possible, at least in peer-reviewed journals.

I always pity the results that are lost because of this; in particular, when reviewers ask for additional research that is feasible in general, but infeasible because the authors are busy with other projects, moving to a different institution, leaving academia, or whatever other reasons. If I were an author in such a situation, I would probably share this reason with the reviewers to explain why additional work is not possible. As a reviewer, I might consider this in my recommendation. My question is: how relevant is this? Does this happen a lot? How likely is it to influence the reviewers' decision? And will the editor care?

These questions are very hard to answer, so good answers might be based on experience rather than hard facts. I would very much welcome those.

Answer 1

The way I see it is as follows: the reviewer's primary job is to judge whether the main claims made in the paper are borne out by the evaluation. The editor's primary job is to fill the conference/ journal with the papers that had the most interesting claims and passed review.

If you send in a paper, and the reviewer complains that additional research is needed, you actually have two options: you can add the research, or you can reduce the strength of your claim. If the editor complains that the claims are not interesting enough, you should look for a less prestigious venue.

So in short, it's perfectly fine to publish unfinished or negative research, but you need to be very honest about it.

Answer 2

It is not a reviewer's business why an author did not expand their research in any of a dozen different directions that invariably good research can be expanded in. Perhaps the author intends to do this in the future, perhaps at this point they are more interested in other research directions, perhaps they are about to leave academia, are suffering from a terminal illness and busy writing their memoirs, or whatever the case may be. None of this information ought to have any effect on the reviewer's decision.

The reviewer's job is to evaluate the actual paper and what it actually contains, and decide if it is correct and good enough research for the venue it is being considered for (or potentially could be made good enough with a small amount of extra work). It is also helpful, but not strictly required, if the reviewer can also offer useful feedback to the author.

In other words, "moving on" is completely irrelevant in the context of peer review.

And as for your "pity [about] the results that are lost because of this", it is understandable, but also irrelevant for peer review decisions. In any case, personally I don't see great cause for concern; researchers are human and lead human lives, with all the complexity and messiness that that entails, and research threads that were not pursued to completion by one researcher will, if they are interesting enough, eventually be picked up by other people.

Answer 3

I think a lot of this is ultimately up to the editor, not the reviewers.

If the reviews are mostly positive but are suggesting additional work that would elaborate on a particular conclusion, the authors could suggest to the editor that they merely add a suggestion for that research in the future and clarify where they are speculating vs. concluding; it would be up to the editor to decide if the paper in this state is still a good enough fit for the journal.

On the other hand, if the additional work is necessary to correct a major flaw or a control experiment to verify the absence of a likely confound, then it seems the authors must either complete the work or abandon the paper.

Answer 4

Appealing to "the graduate student left" or other sorts of "moving on" can be successful, but only in response to some referee requests.

There are two common types of referee comments:

1. To prove your claim X, you need supporting data Y [e.g. added control, missing step in proof.]
2. To make this a better paper, you should do Z. [e.g., study more populations/animals, identify a biochemical mechanism, etc.]

Type 2 also comes in the flavor: "To make this a paper interesting enough for [Journal A], you must do something like Z."

To answer comments of type 1, you must either do what the referee says, or successfully argue that it's unnecessary (e.g. that control is already covered by our data in a different context, etc...).

Though this is not ideal, comments of type 2 are much more common in my experience. And the critical point is that a comment like this are mostly resolved by negotiation between the authors and referees, with the editor making the final call. This is where I think context like "moving on" could be useful. If a referee asked you to do something that would be a good idea, but take two years, that would not be reasonable (and I have seen this explicitly rejected by editors). Even without mitigating circumstances, this is where authors often write, "We believe this is outside the scope of the current paper, and plan to address it at a later time." In the same light, as a referee I would take into account a response like "This would take a major investment of time which is not currently possible because the graduate student now has a real job." There is a cost/benefit tradeoff to some revisions - and the referee does not know all of the costs!

However, as with claiming "outside the current scope," if the referee or editor believes this is truly essential, this can lead to rejection. This rejection is also more likely to occur in fancier journals. ("The graduate student left" isn't going to fly at Nature, or even [Field] Letters, but it might work at your default society journal or PLOS ONE.)