How do academics control their own biases?

Question

As an academic, you are in the business of producing research. I know it might sound odd to call it a "business", but frankly, you get paid to do it, and most of you wouldn't do it if you didn't get paid, hence it is a business and you're selling a product (your research) to your clients (the rest of society) and you get paid a price for that product (grants and aid and donations and other types of support from governments and other institutions).

This means that there is a danger that your research output is driven by a self-serving bias. After all, the 'better' research you produce (as in, research that draws a lot of attention), the more of it you will sell and the more money you will make.

How do academics controls themselves for this bias, that may be occurring consciously or subconsciously?

One might say that peer-reviewing solves this exact problem, but I fear that all peer-reviewing accomplishes is moving the goalpost. Okay, so now the bias of the lone researcher is being controlled, but who controls the biases of the peers? After all, this bias is an industry-wide phenomenon. Your peers, the ones that review your research, have the exact same bias as you do, and are also interested in seeing your research be validated if it brings more attention to your field, a field which is obviously shared between you and your peers.

Hence the bias affects you all at the same time. Who is controlling that? Is there a 3rd party? If not, how do academics, as a group, control themselves?

Obviously, this question is more pertinent for less fact-based fields, so not for example mathematics, where if a wrong theorem is approved, it would be catastrophic and probably discovered soon enough, to the shame of all people involved. But for more experimental fields such as physics, medicine, etc., how do academics ensure that there isn't a cartel of researchers some where all (sub)consciously agreeing to promote research that advances their careers?

Answer 1

Everything is in the end vetted by the community and by time.

Many researchers were fooled by their biases (including Galilei or Millikan who did systematic errors in their studies) and the biases were slowly uncovered by time. This is normal. Of course, a good researcher, as Feynman says, needs to take care not to fool themselves, and good researchers work hard to do so. In the end, however, it is time and the scientific community that filters out the wheat from the chaff.

One key thing to maximize your objectivity is to avoid p-hacking. Be aware what your hypothesis is before you run the experiment. Be aware of what parameters you extract from the experiment and how to factor them into your evaluation of the theoretical vs. experimental model.

Try to have at least two maximally independent ways of getting to a critical result. Understand the assumptions that got you a result.

Taking care of these various things will minimize the probability (it will never entirely go away) that you fool yourself and increase your confidence that you have done a proper job at doing good science.

Answer 2

This is a good question, and arguably I would say research is not without bias. There are many biases that exist, and there are several fields that have actually become too bias and suffer from this cartel mentality, but that is an opinion.

One thing that you can personally do to combat your own bias is to practice being mindful about how you engage with your research and that of others. It’s very hard to remove bias from research because humans are inherently bias. This why there are fields of study dedicated to survey research and making surveys less bias.

You just need to practice good scholarship and encourage/promote it within others. That’s about all you can do to reduce bias in others aside from changing them as people.

Answer 3

The question seems to be about choosing the topic/ type of research so as to pander to grant agencies, rather than falsifying or having experimental biases. I'll focus specifically on the first.

Yes, grants and financial support do sustain research and most of us must tailor our work to have synergy with the funder. But rarely do researchers limit themselves to that focussed research. The usual tendency is to do this directed work (ie the bias in the question) so that other interesting ideas can be supported and pursued in parallel. The origin of dynamic programming and its nomenclature is a notable example of this. So while some research output is indeed directed by the interest of funding agencies, it would be wrong to conclude that all research is driven by this.

Note that the topic directed research could be biased, not its outcomes, unless there is something unethical or inadequately rigorous happening.