10

I have been reading some reflections on research and academia lately, and it has gotten me to think more critically towards what we do, and what the life we are pursuing has in store for us.

Some of you might remember another recent question of mine, where I was inquiring about the right way to set up projects based on an open-letter I read about how academia is not what it claims to be. Now I have stumbled upon this blog entry which focuses on the shortcomings of the single-blind peer review process overwhelmingly used in biomedical research publication.

Now it all brings me back to a disturbing idea that I had some time ago; that the quality of the research and the truth to all we do is essentially hanging on a rather fragile virtue, what one could call "academic honesty". We count on the reviewers and editors objectivity, we count on publishers and researchers naive and good intentions with their work, we count on people not putting their personal benefits before that of the society.

As anyone out on the streets will tell you, we do not live in an ideal world and [call me a pessimist if you will] thus I don't believe in the inherent good of the people (at least I do not rely or count on it). I don't see how academics should be exempt from a degeneration in the society that affects everyone else.

That brings me to my question; what measures exist to ensure this "academic honesty" we seem to rely so heavily on? How do we know/ensure:

  • that the reviewers do not lose their objectivity, for instance when looking at a manuscript of a competitor

  • that there's no collusion between high-profile PIs and editors of "high-impact" journals?

  • that academics (at varying stages of their career) do not consider/prioritize "pushing up" the numbers (e.g. "h-index" or "impact factor" etc) when they set out with their research projects?

  • that grants/prizes/titles are actually given to the better projects/people from the perspective of the greater good, and not based on how well-connected the applicants are? (after all such committees need to have or be composed of other researchers, who else is going to be able to judge the impact and importance of cutting edge research than other researchers?)

I realize that it's a broad question, but I have tried to give a thorough background story to give you an idea about how I got to this idea. Likewise I tried to narrow my concern to one over-arching question (rather than to seek discussion), with a couple of example follow-up questions to make my point clear. I can ask them separately if-need-be, but I think they sit better together, as is.

  • 1
    "that grants/prizes/titles are actually given to the better projects/people from the perspective of the greater good" don't forget jobs and positions. – user4511 Oct 8 '13 at 10:08
  • @VahidShirbisheh of course, I figured it would be implicitly implied, but yes it definitely applies for positions – posdef Oct 8 '13 at 10:47
4

Let me throw in a few points in addition to @Peter Janssons great answer:

There are "tools" that help with some of your concerns

that the reviewers do not lose their objectivity, for instance when looking at a manuscript of a competitor

  • For one thing, most journals I know not only ask for suggested reviewers, but you can also ask that particular persons are not asked to review your paper.

  • The journals I know have at least 2, usually 3 reviews. So a firm "reject" means that several reviewers did agree that the paper is bad. While one may have a biased opinion, with 3 rejects it is maybe time to stop and think whether the paper really is good or not.

  • Bias in peer-review is one thing, but variance is another. And, from my experience, variance is high. IIRC there are studies about marks on texts by different school teachers which were found to vary between good and barely passed for one and the same text. My guess is that peer review is similar.

  • Personally, I'd prefer if peer-reviews were done openly (both sides named, some journals do, though none in my field) as opposed to one side more-or-less blind.

that grants/prizes/titles are actually given to the better projects/people from the perspective of the greater good, and not based on how well-connected the applicants are? (after all such committees need to have or be composed of other researchers, who else is going to be able to judge the impact and importance of cutting edge research than other researchers?)

I think this is fundamentally impossible, because that would imply that it is possible to know which project will be a good project beforehand*. This may be possible for very applied projects, but those would typically be industry projects, not basic research. The "interesting" projects always imply a risk that the thing doesn't work out, no solution is found. Otherwise no research would be needed on the topic.

Sometimes, the huge importance of projects is clear only years (or even decades) after they are finished.

* On the other hand, I think it is possible to find out in advance that a project is poorly designed, so it should not be funded.

that there's no collusion between high-profile PIs and editors of "high-impact" journals?

Of course that can happen. Corruption exists, and I don't think science is fundamentally different from other fields of human professions.
(see below for my thoughts on how to deal with that)

that academics (at varying stages of their career) do not consider/prioritize "pushing up" the numbers (e.g. "h-index" or "impact factor" etc) when they set out with their research projects?

  • In a way we cannot, and IMHO it has to be expected that scientists are intellectually quite able to understand how to optimize a given measure. Thus, the assumption that e.g. bibliometric measures do not influence the measured system will not hold.

  • Whether they actually do it, is another question.

  • There is one very important point: the papers are not kept secretly, you can read them. As a scientist, you can, should and do judge the quality of the papers. IMHO peer review does not allow you to switch off your brain when reading paper - I think peer review is meant to be a sieve that gets the "failed" papers out of the system - while what I want to read are not "barely passed" but "excellent" papers. However, there is a tradeoff, and if you push up the rigour of peer review you'll inevitably throw out also good papers, or papers whose importance will become apparent only later on. That is, too strict peer-review may pose a restriction on the publishing system that disfavors really new ideas, and allows to pass only predictable results.
    However, you won't be able to help realizing who publishes by salami-slicing and who publishes lots of studies with e.g. very low numbers of patients, poorly designed experiments, and OTOH, which groups take the effort to get meaningful numbers of patients, publish on relevant controls, do their homework validating their findings and so on.

  • Also the authors are of a paper are named clearly. And taking together your judgemen of the quality of papers and maybe some citation network tool, you can get an idea who optimizes e.g. citations.

  • Attending conferences and workshops and talking to people you'll get to know people personally so you can judge their character. In addition, you'll hear a fair amount not only of "rumours" and "stories" which also tell about people.

All in all, I think this works as "real life" does: how do you know a company is honest you consider dealing with? You put some advance trust in them, and you use your judgment, including what you hear by your peers about them: e.g. were they recommended by someone you consider reliable.

  • "from my experience, variance [in peer review] is high": On the other hand, it occurred multiple times to me that three or more reviewers gave grades varying by only one point on a scale of about 10 points. – silvado Oct 9 '13 at 11:26
  • @silvado: good to know – cbeleites supports Monica Oct 9 '13 at 16:19
3

I will start by stating, that we probably cannot ensure that any of these problems cannot occur. Doing so will probably mean enforcing structures that are far from democratic and open, it is the downside of the sort of openness we are used to and so we need to learn to live with the system. That, however, does not mean we are without means to work against bad, unethical, behaviour.

Reviewers: The vast majority of reviewers do a good job. Some may be socially challenged to provide criticism in polite ways but have the knowledge to provide critical views. Journal editors will play a vital role in not only selecting reviewers but also screen and evaluate the reviews and provide authors with a balanced view based on the two or more reviews that typically are gathered for each manuscript. The same also applies to foundations evaluating proposals. The review process has two levels where things must go wrong to severely affect a manuscript and the role of editors should not be underestimated.

"Collusion": This problem may be more difficult to spot if the issue originates at "top level". I am not sure I see a patented solution in this case but am convinced that publishers will not support such behaviour and the scientific community will most likely pick up on the problem quickly. There is in other words a social control that will sooner or later make an impact on such problems. This does not prevent them from occurring and since the research community consists of humans we have all the human fallacies found elsewhere in society as well. Removing the illusion that we are of equal moral views and chivalry is a good start.

"inflation": There are good services that calculate, for example, the h-index for you and in a way that others can double check them. I use Researcher ID but such calculations can be done directly in Web of Science and elsewhere. The main point is that by using such a service, the calculation is open and reproducible and it is possible to trace what has been entered. Hence the indicator indices used should preferably be of this kind so that one can double check the value without any effort.

grants: This is similar to the *"Collution" point above. I cannot see any simple solution that would not involve significant costs for the funding agencies (and hence less money given to research. Social control is probably the strongest point here but it also involves openness so that the public has insights into who gets money and why (I am assuming we are talking governmental funding in one way or another; industry and private sources are different)

So I understand and can share some of your pessimism but the alternatives to our current system are in my opinion worse and likely heavier from, for example, bureaucracy. Openness and good guidelines for and continued discussions on ethical behaviour among all are necessary. It is what occurs behind closed doors that starts rumours and may also initiate the problems you describe.

  • Great answer, the only thing I would add is the benefits from being unethical in these ways is pretty small. For example, I unfairly vote for someone who knows influential people to win a prize in the hopes that either my vote, or the person I voted of, influences the influential people in the future to vote for me for something I am not deserving of. – StrongBad Oct 8 '13 at 11:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.