What is the distribution of payoffs in research?

Looking back through history, some research topics can pay off big, eg the double slit experiment in physics. "Pay off" here is not precisely defined, but in this case that was an experiment which started the whole field of quantum mechanics, so that would be intuitively considered as a huge return for the effort invested. On the other hand there are experiments which provide useful or valuable information, but do not appear to lead to any significant advancement of theory; and some of those may require a huge amount of effort.

Are research payoffs fairly evenly distributed (every experiment moves science forward a step) or very unevenly distributed (a few huge successes and many failures)? Is there something like a power-law for the value of research? Is there any relation between the effort or money invested and the payoff?

Also, are there certain properties of a research topic that make it especially likely (or unlikely) to lead to a disproportionately large payoff, such as the establishment of a new theory?

  • 1
    You admit yourself that the notion of payoff is not clearly defined. It is even more difficult to define its distribution, before even thinking of characterizing it. "Success" and "failures" may seem straightforward, but they are not: negative results, although unsatisfactory and very difficult to publish, help other researchers in obtaining positive ones. To summarize, it is a very vague and general question, more likely to trigger a debate on a forum than to obtain a clear answer on a StackExchange community website.
    – Eusebius
    Oct 7 '15 at 7:17
  • @Eusebius: You're right, it's not clearly defined; this is part of the question. But it could be defined in many ways which approximate our intuitive understanding, for example literature-based: number of citations, or total number of descendents in the citation acyclic graph; or rating/ranking-based: with an expert panel that ranks results.
    – Alex I
    Oct 7 '15 at 18:47
  • @Eusebius Just because he didn't define it, doesn't mean the question is meaningless. I could easily provide perfectly serviceable operational definitions, such as Nobel Prize given yes/no (binary), number of times the person is mentioned in popular science books (integer), impact factor or h-index (continuous) and so on.
    – Superbest
    Oct 7 '15 at 20:19
  • 1
    I'll give some half answers which may help you improve the question: Yes, there is obviously a power law, most research has small or no payoff, a small fraction has big payoff. There is a relation between money and effort: With no effort and no money, no science will get done, but just because you have effort and money doesn't guarantee a big payoff or even success. I have an answer elsewhere with my thoughts on the relation between money (also indirectly effort) and success. And yes, topic matters - new topic = more low hanging fruit.
    – Superbest
    Oct 7 '15 at 20:23

There's a vast amount of ill-definition and uncertainty wrapped up in your question... and yet despite that, the answer is almost certainly yes, there is a power-law distribution.

I'm going out on a limb a bit here, because I'm not building on any published analysis that I'm aware of. However, a little analysis of limit cases and fundamental principles can take us a long way here. Let us start with two simple and relatively uncontroversial statements:

  1. Better experimental design leads to better results. It seems self-evident that if you make a bad choice in designing and experiment, it's not going to get you the interesting results you want. At the micro-scale, some choices are clearly better than others, and some are clearly worse.
  2. Sub-fields appear, expand, shrink, and die. As I write this, CRISPR research is hot, and a lot of people are finding interesting results there, and accordingly that field is rapidly expanding. Nobody is doing research on the luminiferous aether because it's been discredited as an idea. Nobody is trying to prove that it's possible to generate machine code from high-level specifications because Grace Hopper did that in the 1950s, when she invented the compiler, thereby initiating what is now a fairly mature and stable research area.

So clearly, no matter how one defines "payoff," any sane definition will see a highly uneven distribution of payoffs both the micro-scale of individual experiments and at the fairly macro level of sub-fields.

Finally, we need to recognize that "significance" is a matter not only of objective value, but also of communication through human social networks. This means that the same result may have wildly different impacts depending on the methods and circumstances of its communication. The history of multiple discoveries in science is ample evidence of this fact; one nice illustrative example is the way in which Barbara McClintock's work on gene regulation was largely ignored until its later rediscovery by Jacob & Monod.

So, we have variation and we have interaction with human social networks, which tend to be rife with heavy-tailed distributions. All of this says to me that it would be remarkable if there were not some sort of power-law distribution regarding pretty much any plausible of definition of impact, significance, and investment. For these same reasons, I think it would also be surprising if one can make any more than weak predictions using this information (e.g., "luminiferous aether research is unlikely to be productive", "CRISPR is pretty hot right now").

And the devil, of course, is in the details...

  • Thank you, this was a valuable answer. Especially "may have wildly different impacts depending on the methods and circumstances of its communication"...
    – Alex I
    Oct 7 '15 at 18:52

Not the answer you're looking for? Browse other questions tagged or ask your own question.