Is it a bad idea to submit a paper merely to elicit reviewers' comments?

Question

Some days ago I was talking with a friend of mine, who is currently a post-doc at the illustrious Italian Institute of Technology (IIT) working on humanoid robotics.

He was quite disgruntled because he recently spent a lot of time testing his latest discoveries and writing a scientific paper to submit to an important artificial intelligence conference, but on the day of the deadline, his supervisor asked him not to submit the paper. Indeed, his supervisor felt the experiments were not good enough, or the results were not excellent.

Anyway, he did not want him to submit the paper, and so he did not do it.

We discussed about it and we both stated that, even if the paper and the discoveries were not excellent, they should have sent the paper anyway. The eventual rejection maybe would have come later, by the reviewers, together with the important review notes. This way, they would have probably got a paper rejection, but they would have been able to treasure the reviewers' review notes. Instead, with his supervisor initial rejection, they won't get anything except their opinions.

So we thought that his supervisor had a bad idea.

My question is: why did his supervisor suggest him not to submit the paper? Why did he not think about the possibility of getting at least the reviewers' comments?

Is a paper rejection so grave a fact? Is it so dishonorable to get a paper rejection?

Answer 1

The other answers mention that submitting a paper only to get some feedback is a waste of time for the reviewers, and it can be particularly damageable for your reputation when the submission is not double-blind. Remember that academia is a small world, and reputation a very important aspect.

In addition, submitting a paper to a venue usually prevents to submit it to other venues, at least during the reviewing process. When you're pretty sure that the paper will be rejected, you might miss other opportunities.

Finally, there is another risk: the paper might be accepted! Of course, at first, you might think it's a good thing, but if the results are not good enough, it might hurt your reputation, and there is of course the possibility that you're wrong. Publishing wrong results (and I'm only talking about mistake, not fraud) is far worse than getting a paper rejected. On top of that, if the paper is published, it might be harder to publish only the improved results, as they might not consist a novel contribution on their own.

Answer 2

Let's start from the latter questions.

Is a paper rejection a so grave fact?

No. Unless you are not resilient enough and ready to become depressed because of a paper rejection. Happens in early PhD, later it usually gets easier to swallow.

Is it so dishonorable to get a paper rejection?

No it is not.

why his supervisor suggest him not to submit the paper? Why did he not think about the possibility of getting at least the reviewers' comments?

I can't know what the particular reasons were, but here are my own personal considerations in situations like this.

1. As a reviewer I value my time. I am very frustrated with papers which are obviously substandard and I have to waste my time to deliver a high quality review. Remember, also low quality papers deserve a high quality review. A review is a standalone artifact too and as such it has to be crafted too. Over time we develop a good feeling what is the required standard for a given venue.

2. My name is connected to a paper when I submit it. You are submitting and delivering a piece of work. Most people with high personal integrity attitude want to deliver only their top pieces of work. I certainly do not want my name to be attached to a piece of work I am ashamed for. Therefore, sometimes I stop myself and don't submit a paper when I know it is not up to scratch. It happens to me quite often.

Notice, as a frequent reviewer, I very well know what the reviewers' response would be. If you are desperate and in a need of a yet another publication, OK, frivolous submissions is a game you can play. If you have a high standard on yourself and your surroundings, playing this game is no good. Delivering good work is what counts, delivering yet another insignificant paper doesn't. In this case we do not speak about rejection on the ground of correctness, or projected significance, but on the ground of not being up to the minimal standard.

Later edit: the above stance is about subjective attitude to things, therefore point 2 applies also to conferences with double-blind review policy.

Answer 3

When I was 16, I had a rather nice idea for a quantum mechanical experiment. At the time, my theoretical foundations were shaky, and I just wanted to know if there were any flaws in it. Unfortunately, I didn't have any contacts in the physics academia, so I had nobody to talk to. Instead, I went ahead and submitted it, and mentioned that I only wanted input. Here's the mail I received:

Dear Prof. Manish Goregaokar,

We have received your submission [id] entitled [title]

Before entering a submission to the reviewing process, we check whether it obeys criteria such as the following:

• Is the topic of research suitable for this journal?
• Does the paper contain original ideas and new results?
• Are the arguments and calculations accurate and correct?
• Is the exposition sufficiently well organized, and worded well?
• Does the overall quality agree with our very tough standards?

I regret to inform you that the editors had to conclude that this work is not suitable for publication in Foundations of Physics.

I would like to thank you very much for forwarding your manuscript to us for consideration and wish you every success in finding an alternative place of publication.

With kind regards,

Gerard 't Hooft Chief Editor

Specific comments from a member of the Editorial Board:

It is not considered a task of the editorial office to evaluate unpublished, or unpublishable, research.

(emphasis mine, I have removed references to the specific paper)

---

If I had not mentioned that I was only looking for input, then I may have gotten a more detailed rejection. However, it is clear from this that the priority of the editors is not to help researchers along, but rather to focus on publishable stuff -- which is perfectly understandable.

Regarding your specific situation (different from mine since you were planning to submit without mentioning that you only wanted an eval): My guess is that the supervisor did it out of courtesy. He did not feel that the editors should waste their time on something which does not benefit them -- sure, helping others is a good thing, but forcing (or tricking) someone into to helping you isn't.

I wouldn't know much about whether or not paper rejection is dishonorable, though.

Answer 4

Reviewers almost universally volunteer their time, and if you are sending a paper that you feel will be rejected because it isn't ready for publication, and/or has serious flaws, then you'll waste the time of a number of people trying to get one or two nuggets of feedback.

Blind review is not the time to elicit feedback on your work; it is meant to ensure a forum for reviewing publishable work. If you want feedback on your work, make the contacts with the right people who will help you, and don't try to abuse the review system by submitting immature work under false pretenses or with an ulterior motive.

p.s. There are places for work that isn't quite ready: conference workshops. Many times they are blind review, but with a high acceptance rate and with the hope that workshop attendees will provide good feedback on the work and it can be improved for conference or journal publication.

Answer 5

Paper rejections are far more common than one might think. A rejection rate of 50% is not uncommon in most average journals that I know about. In some high-reputation journals it may be over 90%. So rejection is not uncommon. There are of course different reasons for a rejection, some worse than others (incomplete and in approximately falling order): the manuscript (MS) content is not appropriate for the journal; the MS (and/or figures) is poorly formated (to the point that it is hard to follow); the MS has severe language problems; the science is poorly supported by references and/or the discussion; the MS contains scientific errors or misconceptions; and then more shady problems such as falsification and plagiarism.

Now to send in a MS just to get comments from reviewers. As an editor, I really would resent this behaviour. After all what is happening is that the Ms would take up at least one editor and a couple of reviewers (free) time (none are usually paid for the work). There are authors who send in their half-baked manuscripts just to have reviewers help them iron out things they were too lazy or incapable to do themselves. I know such cases personally. The risk is that the paper gets rejected but if not then someone pours in a lot of work for very little credit (which in turn is taken up by someone else).

So from the description above, I would say that the advisor may have done the honest thing and wanted to prepare the paper as best he/she could, you be the judge. It is of course a grey zone when something may be good enough to become accepted for review or just not good enough and head for rejection. Where one draws this line is perhaps personal but it is far better to stay away from the grey zone altogether. I would therefore suggest that discussing this matter with the advisor may prove to be both a fruitful and interesting endeavour.