Where do researchers get the money to replicate others' work?

Question

I'd like to follow up on my previous question. If scientists have to replicate and validate/refute their peers' research, where do the "provers" find funding?

In my 2 years as an undergrad at a public American university, I've heard of a lot of professors researching new things and/or trying to find funding to research new things, but I've never heard of a professor dedicating resources to replicate his/her peer's work.

As I explore existing literature for my research project, I see that I could have used existing literature to guide the experiments I did for my project. Along the way, I could have replicated + validated* the existing "new" methods before trying my new method. Is this how researchers usually review their peers?

That is, professor X gets a grant to explore field A. Professor Y comes up with model 1 for field A. Using his own funding, Professor X designs and conducts an experiment to replicate model 1 before testing his own model, model 2.

Note: I assume my question depends on the field of study since a computer scientist can more easily verify a model + experiment than a chemist.

*Is there a better verb or phrase for replicate + validate/refute? Or does validation imply replication?

Answer 1

In my field (cognitive science), your supposition that replication is primarily pursued by investigators prior to extension is true. For example, I have an idea while reading a paper on learning targeted movements, I first implement the paradigm as described (either an exact replication or a conceptual replication) and then I extend it to test my idea. Thus, the money for this comes from the same places that the money for all research comes from: largely from government allocations and grants, partly from NGOs and industry.

However, there has a been a recent trend--again, in my field, but as part of a larger movement in the social sciences--to improve replicability of results. This has included fostering availability of code for software projects and pursuing independent replications purely for their own merit. Last semester I attended a talk on a set of these replications, which, along with similar attempts from other labs, were pushing to overturn the original result. In these cases, the money to pursue the replication is probably somewhat independent from other projects of the researcher. This means funding agencies would be unlikely to provide a grant entirely for this work, so it is probably going to be supported as a passenger on previously funded work or on University funds (e.g. start up and salaries).

Answer 2

I can only tell you from the field of chemistry (and for germany). Normaly you only get funding from institutions to produce/research new stuff. Additional you get money (resp. Lab space, chemicals, technicians, positions for phds/post docs) from the university/institute to work on your field.

So how to check others work. Thats easy. Lets say professor A finds a great new reagent to produce something. If you work in a similiar field or you (resp. a lot of times the phds get the ideas and the work) wants to get something different but you want to try his reagent/method you will redo his work to produce somthing new with it.

That means you use his work (and you will hoefull cite him for that) to try some of your ideas. And then you will also see, if what he published was correct.

For the down side (in my expreience),especially if the results were posted in a top tier journal it is very hard to reproduce others work at first. This may be because of the lack of your knowledge, but oftentimes authors "forget" to put in some details/knacks which will make the whole reproducing very hard. This will give Prof. A some more time to use his published method for his own research befor a colleague can copy his approach. Oftentimes something published in a mid or lower tier journal is better to reproduce since the experimental section is oftentimes more detailled or more "realistic". Please note, that this paragraph is only from expreience and I don't want to blame someone special. It`s just what I experienced.

As an addition, I think there a (few) journals/book series where the reviewer should/must reproduce the result first befor he can accept your paper. but that's really rare and I right now can't list you one from my memory.

Answer 3

In human (and non-human animal) behavioral fields such as Psychology and areas of Medicine, which often involve many paid participants and quite a lot of expense, replication is often NOT done "first" or separately from a new experiment! There just isn't the money and time to be so inefficient - so you need to dual-purpose and build upon existing literature instead.

Story Time

I'll use the example that was used to explain it to me, because I think it's a great example of the concept. In the study Effects of Marijuana on Memory (Weil, Zinberg, & Nelson, 1968), the researchers effectively wanted to know if pot screws with your memory - does it make you forgetful, etc? So they designed a simple experiment where they gave some people real pot, and other people fake pot - then they tested their memory. In that experiment, it turned out that the answer was "pot does screw up your memory" - people have a decreased ability to recall things while high on the ganja.

But that was just one small experiment - what if it was just wrong? As a part of the idea of building upon past work to answer more questions, they designed a new study that asked if the effect of pot on memory depended on prior experience with the drug. This is a new question, but note that it assumes that pot has an effect on memory in the first place! In the new experiment they split people into the previous two groups (pot and no pot), but then split those groups as well so they were testing experienced pot heads and people who hadn't smoke it before.

Part of analyzing the results includes comparing the memory to pot heads and non-drug users, as well as comparing the memory of people who did and did not smoke marijuana. This is, effectively, a replication of the previous study! However, the new study made it possible to look at interaction effects - and it happened to turn out that pot only made pot-naive people forget things they encountered while high. True pot heads, it would seem, have the same memory stoned as sober - and stoners sober had no notable difference in memory to non-stoners. Go figure!

If the first time was interesting...

The key in such experiment designs is that not only is a design testing something new (and thus making it worthy of funding and publication in most venues!), but it also serves to re-test (and replicate) previous studies. And this happens pretty darn often!

If the effect is interesting, it naturally raises a lot of other questions - questions that are closely linked to the original effect. In testing these other questions, it is often difficult to even avoid testing the original thing too, and so replication is a natural part of the scientific process in these fields. It also produces a lot of the "controversy" that some fields experience.

My favorite example of this is probably gender/sex differences in intelligence factors. Do men have better visual-spacial reasoning? Do women have better verbal abilities? For years it was thought that these were real, significant effects and gender differences - and more recently they appear to be very close to pure fantasy, with many studies indicating no difference in men and women whatsoever in such areas. At most, it is estimated that both of these effects have the respectively-weaker sex at average outperforming 30-40% of the opposite sex - rather than the 50% expected if the effect were nonexistent, with many studies indeed showing less, no, or reversed differences.

And none of these studies had to be pure replications - any study that looked at IQ needed only to collect basic demographic data from the participants to allow such replications, which they probably were going to get anyway. No extra funding, yet loads of replication!

Answer 4

I really wish explicit replication of high profile studies was a priority for funders. However, as was mentioned there isn't typically funding for this kind of research.

The state of social science research is pretty terrible. I have tried to replicate results of past studies en-route to designing/analyzing my own studies. At times it can be like seeing that the emperor wears no clothes.

I think a revolution is coming though. Open-source is becoming more common, and I think you'll see a younger generation of researchers, like myself, at the very least opening our code up to the public or other researchers. As an aside, if you want to see a researcher squirm, then ask for their code. Better researchers will probably move away from point and click stat programs, so that the entire process they went through en-route to results will be transparent. In addition, there's a slow revolt occurring against null-hypothesis testing, which has really been a major factor in the replicatability crisis, and you might see more funders putting money explicitly towards Bayesian models for common research problems where the evidence seems mixed or the policy results of the research-evidence are minimal.

Answer 5

For impact evaluation there is funding from the 3ie replication project. They list a number of studies they would like to see replicated. Expertise needed is development economics/epidemiology/medicine/development related political science or maybe statistics.