Is there research on the extent to which complexity of the proposed method(s) impacts the acceptance rate?

Question

I heard this speaker at IJCAI complained that his paper got rejected because the methods he used were too simple despite the results being highly competitive (twice the state of the art, at least according to one metric). The complain can be listened in this video at 29'30''. Excerpt:

(Addressed to reviewers) Accept simple papers if they demonstrated what they were trying to show.

It wasn't the first time I heard such a complain, as simplifying existing solutions is arguably a contribution, yet sometime despised.

Is there any research/study/survey that looked at the impact of the complexity of the method(s) proposed in a submitted paper on its acceptance rate?

I am mostly interested in the field of computer science > machine learning / NLP / data mining, and English-speaking venues.

Answer 1

I haven't seen any research on the correlation between the complexity of the methods and the acceptance odds of a work and I think it is relatively hard to do a study on this. First because being simple or complex is completely subjective and domain dependent and hard to quantify. One can not generalize a pattern that is seen in a particular scientific community to others. Moreover, the information about rejection of papers is not publicly or easily accessible making the hypothesis hard to investigate.

That being said, in computer science machine learning community, I have seen that people are pushing towards critiquing the unnecessary complexity of the suggested methods and putting more emphasis on the actual results rather than the simplicity or complexity of the methods. There are many simple methods that are being published because they actually work well. I personally don't think that simple ideas which show good results are necessarily prone to rejection, but the opposite argument (complicated methods having higher chance of acceptance) is sometimes true.

On a related note, explanation of the ideas and presenting a work in the most simple way for people to understand, is being emphasized more and more. One reason of people trying to complicate things is that they think the less it is easy for reviewers to understand their work, the higher their chance of acceptance. I personally think that ideas in the papers should be explained simply and clearly because if it can’t be explained simply, probably it can be done better. And reviewers should not praise the work they don't understand, because it is the duty of the writer to explain everything simply enough so that the corresponding community understands it well.

Answer 2

This is relevant to the Kuhn - Popper debate about the way science works, and there is a considerable literature here... I don't know about statistics, but I do know this is extremely common! A new simpler approach will usually be dismissed out of hand with scant regard by the powers that be with their strong ties to the traditional model.

Kuhn notes that researchers get locked into Paradigms. Those in positions of power (referees and editors, professors and deans) prefer papers that continue their work, use the tools they invented or are familiar with, cite their journals or journals indexed by their favourite citation company. There is also a factor of workload in relation to interest: They skim things quickly and pay particular attention to headings, tables, figures, equations and references - looking for things that connect to them/their interests/their journal. Some reviewers show no evidence of having actually read the paper, particularly for so-called 'top journals'. If they are not interested, or they have decided it a 'poor approach' a priori, then it doesn't matter how good the approach is or the results are, and you will be flipped off with comments about the style, format, equations or references (e.g. a complaint of lack of references to X or insufficient or unnumbered equations, or lack of road map glue telling them there is are Introduction, Methods, Results and Conclusion sections). That is there are not only paradigms within fields, in relation to how you tackle a problem, or the theoretical framework or model you work in, there are paradigms in relation to modes of presentation.

Popper on the other hand espouses Parsimony and Refutation. The simpler theory or model is better, other things being equal. A poor theory grows more complex as it keeps being adjusted or extended to deal with new cases that cause it difficulties. Usually the poor theories and models don't die off until their proponents and perpetrators do, until they get so complex as to be completely unmanageable and they eventually topple over and fall into oblivion. Conversely, a simple model will be ignored until it is demonstrated that it handles everything that the older theories tried to do, and makes new advances and predictions that are borne out. Popper's ideal researcher is totally different from Kuhn's dogmatic paradigmatic researcher. A good researcher is making predictions into the unknown where the different theories predict different outcomes. A good researcher is not trying to bolster their models, but rather to refute them - find the holes rather than plug the holes.

So how do you deal with this? A very good question - glad you asked...

You have to face it head on. Choose publication venues that are high quality but have the kind of format and expectations that allow you to present your new simpler ideas and models, even though it may not be fully worked out and compared against all your thousands of competitors and their thousands of datasets or examples. Get feedback, and find who is sympathetic, where they publish/review/edit etc. Eventually, you need to target the archival traditional journals, the bastion of the current paradigm, and fit things into their mould, follow their rules, explain the equations/models you are competing against in detail, especially those that are pushed by the journal and its editors/authors (at least they get a citation). Clearly point out the advantages...

• The theory explains more with less (effectiveness/parsimony)
• The model/algorithm is shorter and/or runs faster (efficiency)
• The results are more accurate and/or have lower variance (efficacy)