17

I find it very annoying as a reviewer when I get asked to review a paper several times as it gets submitted to different journals/conferences or, worse still, I help send it to the reject pile only to have it published elsewhere with my comments ignored. I definitely get the impression that a small minority of authors "shop" their papers around too much, which wastes a lot of the community's time. Often these papers are completely unpublishable, but sometimes it seems the authors are just going down some list until they reach the appropriate "level", rather than submitting appropriately in the first place.

I often devise hypothetical schemes to prevent this. For example:

  1. Require authors to pay to submit.
  2. Require authors to disclose the submission history their article.
  3. Have journals/conferences publish a list of rejected articles.
  4. Calculate an acceptance ratio for authors, along the same lines as bibliometrics.
  5. Somehow publish rejected articles too, so that they cannot be published elsewhere. E.g., have a journal that accepts nearly 100% of submissions, but includes a review or rating with each article, so readers know which ones are the good articles.

Of course, these are just wild ideas, many of which are impractical and would greatly annoy authors. I'm curious to know if any such schemes have actually been tried and, if so, whether they seem to have worked. Surely, in some far-flung corner of academia, one of these ideas has been tried.

Question: Are there systems that have been tried by journals/conferences to disincentivize authors from submitting their articles to too many journals/conferences? If so, did they work?

I am only aware of one experiment along these lines: ICLR - the International Conference on Learning Representations. All submissions and all reviews are made public and they stay online even if they were rejected. I don't know what effect this system has had. I would like to hear other examples and also know if there is any evidence regarding the effect of such policies.

  • 9
    Take a cue from the exponential backoff algorithm used in TCP, etc: when you are sent the paper to review for the nth time, sit on it for time 2^n before sending in a report recommending rejection. (HHOS) – Nate Eldredge Aug 9 '18 at 23:35
  • 4
    I think there is a big difference between shopping around an article which is unpublishable/wrong/highly flawed, and shopping around an article because it's often in the authors interest to publish in the highest 'level' journal. Why should authors be penalized because an editor or reviewer didn't perceive their manuscript as meeting some vague and fluid criteria for 'interest' or 'importance'. Maybe the solution is not to have such a finely graded hierarchy of journals with quite arbitrary differences between the different journal levels. – Bobgom Aug 11 '18 at 4:08
  • 6
    The issue is that publishers have setup elaborate hierarchies of journals with often ill-defined criteria for what does and does not meet the required level of impact or importance. If the editors and reviewers time is wasted, its because of the existence of such a hierarchy in the first place, where apparently authors are meant to simply know whether a paper is in keeping with the level of 'impact' or 'importance' required by the publisher. Why is it the authors job to decide whether their paper is in keeping with the brand of a particular journal. – Bobgom Aug 11 '18 at 19:43
  • 3
    I was in at least one university workshop where shopping papers was explicitly recommended as a standard practice and a best-practice. I thought it had a bad odor. (N.B. I was the only STEM person, with everyone else more experienced humanities staffers, none of whom objected.) – Daniel R. Collins Jul 9 at 2:21
22

A version of your suggestion 1. is implemented in my academic field (mathematics).

In mathematics authors usually pay dearly to submit to any particular journal, not in money but in time. Namely, an unacknowledged but hard to dispute stage of the refereeing process in mathematics is the part where the referee lets the submission sit on their shelf for N months. It would be interesting to track the dependence of N on various factors, but based on my own experiences as an author (and okay, yes, as a referee!) I believe the median value of N to be about three months, with a higher mean. Especially, when I submit a paper to a large range of reputable journals in my field, I feel that I am incurring a risk -- small, but not negligible; let's say 3-5% -- that the referee will wait a year or so before rejecting the paper with a wholly superficial accompanying report (say, one paragraph that does not make specific reference to anything in the paper past the introduction).

This makes it rather impractical to shop a mathematics paper around to a large number of journals at about the right level. (If you send a less than excellent paper to a truly excellent mathematics journal, the chances are good that it gets bounced back to you quite quickly. I have heard stories of the absolute top journals in mathematics keeping a paper for a year or more and then responding negatively...but then I have also heard of them doing this and then accepting papers that I would have guessed they would reject.) This certainly works for me: I have submitted over 50 distinct math papers for publication. Most often I step down in quality after the first rejection and always after the second rejection. For me, having three or four journals consider the same paper feels like quite an ordeal....because it is likely spread out over several years!

Maybe this sounds sarcastic. It is not intended to be. If anything, it is a relief to think that there might be some good effect of the current endemic slowness of mathematical refereeing. Because we have a thriving preprint culture, it is hard to argue that slow publication has a significant effect on the intellectual progress of the field. If only the slowness were equidistributed, I think there would be little to complain about. (But it isn't, and I suspect that my suffering is as nothing compared to that of some of my junior colleagues...In fact, just now I realize that while for many years now my papers have been treated better by top journals whether they take them or not, at the beginning of my career I had several papers languish in top journals for a really long time. The rich should have to pay more, not less.)

  • 10
    There's another sense in which the "rich pay less": For young people trying to get jobs, promotions, tenure, etc., it can be very important have a paper not only available on the arXiv but officially accepted by a respectable journal.For old folks like me, much less damage is done by a referee who ignores my paper. Sure, it's nice to be able to put "accepted" on my CV and on my annual report to the department administration, but nothing disastrous happens if "submitted" lasts for a couple of years. – Andreas Blass Aug 11 '18 at 3:52
  • @Andreas: I agree, and in this regard I am also "old like you." – Pete L. Clark Aug 11 '18 at 15:34
11

When a paper is rejected, there are two possible reasons:

  1. The paper is bad
  2. The review is bad

The majority is 1. (hopefully), but 2. is also very common. Some random examples:

As there is no way to distinguish 1. and 2., if their authors resbumit, should we (i) punish both or (ii) ignore both?

Require authors to disclose the submission history their article.

An example: see the Reviews from Prior Submissions section of the CFP of Oakland, one of the big 4 in security.

  • 2
    I like your analysis. Just a small observation: as authors are also reviewers it seems very unlikely to me that the majority is 1. I suspect 1 and 2 to be equal. Even beginning authors start mostly under supervision of experienced authors. Reviewers overestimate the quality of their review. – user93911 Aug 12 '18 at 6:15
9

I'm not aware of any disincentives. However, I can describe a process we're currently undertaking to preserve the valuable information that is often lost when a manuscript moves from one journal to another. I consider the present situation wasteful of time, effort and resources. I participate in the Editorial Board of a general medical journal considered the premier medical rag in my country. At the present time, we're running an experiment of sorts in collaboration with other medical subspecialty journals and craft groups to share submission histories, reviewer comments, manuscript revision and decision histories. This will enable us to track a submission, at least within the consortium.

We were surprised that editorial processes haven't been subject to the rigorous experimental design that we ourselves expect from our manuscripts. Thus, we decided to design one ourselves. In this experiment, the author is free to choose to which journal he or she submits. If an accept decision is eventually made (regardless of the number of revision rounds), then there is no further action on the consortium's part. However, if a reject decision is made (again regardless of the number of revision rounds), then the manuscript's meta-info are logged onto the consortium's database. If the author chooses to submit to a member of the consortium, then the manuscript's history is made available to the editorial team of the second journal. And so on.

There are flaws in this idea, of course. Journal coverage is incomplete, for instance. We will not have access to histories from journals outside the consortium. Some of the previous decisions may colour subsequent editorial decisions is another potential concern.

That's the general idea, at least. We're in the early stages of its development.

I do hope that this helps.

  • Thanks, that's very interesting. I'm keen to hear how it works once deployed. The practicalities of sharing information between journals is a major issue. – Thomas Aug 10 '18 at 0:25
  • That's nothing compared to getting coffee orders from several silverback Editors-in-Chief during the consortium meetings. – St. Inkbug Aug 10 '18 at 0:42
  • Presumably the meta-info that you share is such that authors can't cheat by the trivial tactic of changing the paper's title. But do you need to worry about more sophisticated attempts to cheat? – Andreas Blass Aug 11 '18 at 3:56
  • @AndreasBlass Yes. At the moment, it's not designed to detect cheating, although I'm sure if it survives the pilot phase, these features might be added on. – St. Inkbug Aug 11 '18 at 8:21
4

I can’t help noticing some contradiction in your post. You observe a small minority of the authors shopping around and are concerned about the uneven communities’ waste of time.

Your main problem seems to be your personal annoyance, especially seeing a paper being published without your valuable comments (given while rejecting it) being addressed.

By now you probably have your list of this small minority of authors in your field which ‘shops around’. You are certainly not obliged ‘to help’ these authors’ papers to the pile reject. In addition I follow BruceET here. Just don’t review a paper you previously rejected. It will save you a lot of time.

Your suggestions (the legal ones) are interesting as a mind experiment. In addition: as bad reviewers are just as annoying as bad authors I (solely as a mind experience) would suggest the following: let each review be accompanied by statistics on the reviewer.

  • the reviewer’s field of specialism, years of experience, number of publications published as first author and the journals published in.

  • the reviever’s review history (number of review reports, their length, the journals reviewed for, duration of the reviews, reviewer’s decisions made in comparison with final editorial decisions).

  • of course, as you propose for authors, these reviewers’ statistics are shared with the journal’s consortium.

In addition we could think of similar statistics for associate editors.

To continue:

  • after an editorial decision, authors are requested to provide a review on the quality of the review reports (validity, coherence and structure) and editorial guidance and responsiveness. These scores will be added to the reviewers’ statistics mentionned above.

In this way we can more easily identify bad reviewers and bad editors. This may certainly help authors, reviewers and editors to maintain the quality of the review process.

3

There are two important perspectives here.

Author's point of view. Journals have standards of various levels, and 'topics of interest' can change in important detail over time from one chief editor to the next.

I think it is entirely reasonable for an author to submit to the highest-level journal that seems realistic. In my field, rejection will usually be very quick if the paper is far below standard for the journal.

If an associate editor feels the paper has some merit, then it will go to review. If reviews are positive it may be accepted, provisionally accepted pending suggested/required revisions, or rejected (due to the editor's taste or too high a backlog of 'worthy' papers). If the paper is rejected, one can hope that useful suggestions for improvement will be included. Also if reviews are unenthusiastic, the paper will be rejected. Then it is up to the author to get another idea or to re-submit elsewhere, possibly with some modifications.

I have served on publications committees of a couple of societies publishing journals in my field. Serious efforts have been made to speed up the review process. Some major journals now make final decisions on 75% of papers within six months. I detect no deliberate attempt to use delay as a penalty for successive submissions.

I see no reason an author should take one or two rejections as a sign to abandon the essence of a paper. However, feedback should be used to prompt revisions, and after a couple of strategic downgrades in the prestige of journals submitted to, perhaps it is best for an author to move on to other work. These days there are dozens of predatory 'journals' that will accept almost anything, and self-respecting authors will protect their reputations by avoiding them like the plague they are.

Reviewer's point of view. Reviewers should set reasonable, but generous, limits on the number of papers they agree to review and try to respond quickly (the task will not get easier if delayed).

My view is that the only proper response to a request to review a paper for which one has previously recommended rejection is to refuse immediately to review it again, saying why. That will take almost none of the reviewer's time. Thus the reviewer can spend precious time giving a helpful review of the the next relevant paper to come along.

The author deserves an unbiased, fresh review process. However tempting it may be to believe in one's preeminence, few reviewers are uniquely qualified to pass judgment on any one paper.

2

I find it very annoying as a reviewer when I get asked to review a paper several times as it gets submitted to different journals/conferences or, worse still, I help send it to the reject pile only to have it published elsewhere with my comments ignored.

I believe you are taking excess ownership in the outcome. It's not your job to set yourself up as the ultimate gatekeeper of the entire scientific literature. As a reviewer, you are an extra pair of eyes (perhaps with some especial subject matter familiarity) to help the editor make a decision. But he still has full ability to ignore your advice and accept a paper.

And authors have the right to not make changes they don't agree with, knowing full well that this may/not lead to rejection and movement to another venue. And to pull papers from journals pushing for changes they disagree with.

It's the author's paper, hence the byline. And the editor's magazine. Your position is much more limited and advisory than theirs.

Of course it's annoying to see the paper again, but in that case, I recommend you to just tell the new editor that you previously rejected the paper and won't review it again. Don't send them a copy of the old review or justify your reject recommendation. Just pull yourself from the process. No more cause for annoyance then. (This also gives the author a fair chance from different eyes.) Of course, the simple note that you saw it before and refuse to review it may influence the editor. But you should not worry about that pro/con (if he wants to give the paper a fresh chance or just gets rid of it).

  • I strongly disagree with the attitude that I should not review a paper a second time. It might be hard to find another suitable reviewer. And, if I've read the paper before, it takes fairly little effort to review it. If some other person gets asked to review, they will need to expend more effort. Perhaps your experience is different, but in my field reviewer exhaustion is a serious issue; that is the crux of my question, which you are not addressing. -1 – Thomas Jul 10 at 19:21
0

Unfortunately, the process user St. Inkbug suggest, might be illegal. According to EU, sharing info without the approval of users.

Somehow, I need to agree with number 5.

Somehow publish rejected articles too, so that they cannot be published elsewhere. E.g., have a journal that accepts nearly 100% of submissions, but includes a review or rating with each article, so readers know which ones are the good articles. and it is now done by Sci, Sci — Open Access Journal http://www.mdpi.com/2413-4155/1/1/1/htm

  • 3
    If the EU prohibits sharing information without the approval of users, then I"d expect the journals to require users to give that approval as a condition for submitting a paper. – Andreas Blass Aug 11 '18 at 13:13
  • 1
    @AndreasBlass thank you for a comment, in this case, the information is trade secret, and it is protected under the clause of Information act from 1997 and 2008, also consumer protection act regulate this, since parties didn't engage in any transaction(in this case publishing) This is why I think the proposition could be degerous. – user96746 Aug 11 '18 at 14:26
  • Lawyers are involved. – St. Inkbug Aug 11 '18 at 22:19
  • @St.Inkbug I dont understand? – user96746 Aug 12 '18 at 4:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.