3

For me, publishing results only when they are final, complete, and positive is more of a tradition than a necessity. I would find it valuable if others were able to publish what they have tried, even if they have not achieved what they (initially) aimed for. Unfortunately, this is very often not possible, at least in peer-reviewed journals.

I always pity the results that are lost because of this; in particular, when reviewers ask for additional research that is feasible in general, but infeasible because the authors are busy with other projects, moving to a different institution, leaving academia, or whatever other reasons. If I were an author in such a situation, I would probably share this reason with the reviewers to explain why additional work is not possible. As a reviewer, I might consider this in my recommendation. My question is: how relevant is this? Does this happen a lot? How likely is it to influence the reviewers' decision? And will the editor care?

These questions are very hard to answer, so good answers might be based on experience rather than hard facts. I would very much welcome those.

  • 4
    If the reviewers find that the work is not worth publishing without additional things, why would they care why the author did not and does not plan to do them? That will not change that they find the work incomplete. – Tobias Kildetoft Jan 19 '17 at 12:21
  • 3
    Depending on the situation, the reviewers may be satisfied if you simply identify the additional research as a possible "future direction". They may just want to make sure that you considered the additional research, even if you chose not to do it. – mhwombat Jan 19 '17 at 12:42
  • @TobiasKildetoft I have edited the question and included "As a reviewer, I might consider this in my recommendation." I am unsure how I should and would consider this exactly, though; this is part of my question :) – bers Jan 19 '17 at 12:43
6

The way I see it is as follows: the reviewer's primary job is to judge whether the main claims made in the paper are borne out by the evaluation. The editor's primary job is to fill the conference/ journal with the papers that had the most interesting claims and passed review.

If you send in a paper, and the reviewer complains that additional research is needed, you actually have two options: you can add the research, or you can reduce the strength of your claim. If the editor complains that the claims are not interesting enough, you should look for a less prestigious venue.

So in short, it's perfectly fine to publish unfinished or negative research, but you need to be very honest about it.

  • 5
    Well, in my experience (math), the editor asks the reviewer for their opinion on the importance / interest of the results. It would not be uncommon for a reviewer to say something like "the results currently claimed in the paper are correctly proved, but are not interesting enough; to reach the necessary level for this journal the paper should answer the following stronger question..." Of course it's up to the editor to agree with this. – Nate Eldredge Jan 19 '17 at 15:23
  • @NateEldredge That's a good point. The editor usually uses the advice of the reviewers to assess novelty. But I would say it's not the primary role of the reviewers. – Peter Jan 19 '17 at 19:42
  • 4
    Some journals have a preliminary review stage in which the reviewers are only asked to assess novelty, significance and interest. If their responses convince the editor that the paper would likely be acceptable if its results are correct, then there is a second, longer review cycle to determine its correctness. – Nate Eldredge Jan 19 '17 at 19:45
  • @NateEldredge I did not know that. I guess it's sometimes more efficient to assess novelty first. – Peter Jan 19 '17 at 19:48
4

It is not a reviewer's business why an author did not expand their research in any of a dozen different directions that invariably good research can be expanded in. Perhaps the author intends to do this in the future, perhaps at this point they are more interested in other research directions, perhaps they are about to leave academia, are suffering from a terminal illness and busy writing their memoirs, or whatever the case may be. None of this information ought to have any effect on the reviewer's decision.

The reviewer's job is to evaluate the actual paper and what it actually contains, and decide if it is correct and good enough research for the venue it is being considered for (or potentially could be made good enough with a small amount of extra work). It is also helpful, but not strictly required, if the reviewer can also offer useful feedback to the author.

In other words, "moving on" is completely irrelevant in the context of peer review.

And as for your "pity [about] the results that are lost because of this", it is understandable, but also irrelevant for peer review decisions. In any case, personally I don't see great cause for concern; researchers are human and lead human lives, with all the complexity and messiness that that entails, and research threads that were not pursued to completion by one researcher will, if they are interesting enough, eventually be picked up by other people.

3

I think a lot of this is ultimately up to the editor, not the reviewers.

If the reviews are mostly positive but are suggesting additional work that would elaborate on a particular conclusion, the authors could suggest to the editor that they merely add a suggestion for that research in the future and clarify where they are speculating vs. concluding; it would be up to the editor to decide if the paper in this state is still a good enough fit for the journal.

On the other hand, if the additional work is necessary to correct a major flaw or a control experiment to verify the absence of a likely confound, then it seems the authors must either complete the work or abandon the paper.

3

Appealing to "the graduate student left" or other sorts of "moving on" can be successful, but only in response to some referee requests.

There are two common types of referee comments:

  1. To prove your claim X, you need supporting data Y [e.g. added control, missing step in proof.]
  2. To make this a better paper, you should do Z. [e.g., study more populations/animals, identify a biochemical mechanism, etc.]

Type 2 also comes in the flavor: "To make this a paper interesting enough for [Journal A], you must do something like Z."

To answer comments of type 1, you must either do what the referee says, or successfully argue that it's unnecessary (e.g. that control is already covered by our data in a different context, etc...).

Though this is not ideal, comments of type 2 are much more common in my experience. And the critical point is that a comment like this are mostly resolved by negotiation between the authors and referees, with the editor making the final call. This is where I think context like "moving on" could be useful. If a referee asked you to do something that would be a good idea, but take two years, that would not be reasonable (and I have seen this explicitly rejected by editors). Even without mitigating circumstances, this is where authors often write, "We believe this is outside the scope of the current paper, and plan to address it at a later time." In the same light, as a referee I would take into account a response like "This would take a major investment of time which is not currently possible because the graduate student now has a real job." There is a cost/benefit tradeoff to some revisions - and the referee does not know all of the costs!

However, as with claiming "outside the current scope," if the referee or editor believes this is truly essential, this can lead to rejection. This rejection is also more likely to occur in fancier journals. ("The graduate student left" isn't going to fly at Nature, or even [Field] Letters, but it might work at your default society journal or PLOS ONE.)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.