Results that are too good to be true - peer review

Question

So, I'm reviewing a medical study (open-label trial) that compares the efficacy of different drug doses on a patient population (heart failure). The study is arguably of low-quality, compared to the landmark trials that established the benefit of this drug in this population: low sample size (~100 compared to 2000), short follow-up duration (~3 months compared to 16 months), open-label and without placebo.

The results the authors report are too good to be true - the study arm that can be directly compared with previous trials on this topic (same population, same drug dose) had a 4 times greater reduction in NT-proBNP (heart failure biomarker) than in the original study! And remember! Only in 3 months compared to 16!

Furthermore, during previous peer review, another reviewer suggested that it is a study limitation that no markers of functional capacity were available (6-minute walk test), and the authors just included it, seemingly out of thin air! (It wasn't previously mentioned in the methodology of the study).

Another indication that their data is falsified is that they report that all their measured variables were normally distributed - in my experience with similar variables, they are normally log-normal, not outright normal! (Although there is no proof that's always the case in literature).

The editor is seemingly hell-bent on publishing this paper, as it has undergone 4 rounds of review, and reviewers that reject it are being replaced one-by-one.

I know I can't reject this paper on the strong suspicion of foul play - what is the right way to tackle this?

EDIT: Thank you for your responses and comments. The authors ascribe the discrepancy to a different sample make-up, but it's (subjectively) too great to be simply due to the sample composition. As for anonymity, it didn't occur to me initially - I'll try to maintain enough ambiguity as to prevent a breach of blinding.

Answer 1

In a comment on @Buffy’s answer, you wrote:

[…] my thinking is that it would be wrong to reject these researchers purely on the suspicion of falsified data - after all, I could be wrong.

There is a misconception here that is worth pointing out. The key thing to remember that it is fully the authors’ burden to convince you that the research is correct in order for you to recommend acceptance of their paper; it is not your burden to prove that some suspected flaw you are perceiving is real before you are allowed to recommend rejection. In other words, rejection should be the default decision for any research that doesn’t meet high standards of rigor and address any reasonable criticism that might occur to a referee. So for example, if you thought there was a 25% probability the research results were unreliable and a 75% probability they were correct, then recommending rejection (or at least a revise and resubmit to allow the authors to fix the flaws you are pointing out) would be the correct decision, even though it would still be the case that “after all, you could be wrong”.

Following this logic, if the results really seem too good to be true, then it’s up to the authors to convince you that they aren’t too good to be true, by increasing their sample size, shoring up any methodological deficiencies you point out, and/or rebutting your critique about normal versus log-normal distributions. Consider giving them the chance to do so.

Being skeptical does not mean you are saying the results definitely aren’t correct, you are simply saying you’re unconvinced and that you don’t think the paper should be published until it can more rigorously defend the claims it is making. As @Buffy said, simply state your honest opinion — that is precisely your duty as a reviewer.

Answer 2

Say to the editor what you say here. You can't prevent the publication, but you can be honest. Even if there is no foul play, if the results are anomalous then there are probably methodological problems, such as sample size.

You can recommend rejection. If the editor publishes anyway and takes you off the list of reviewers, you are probably better off. Tell it like you see it. The responsibility is with the editor.

Answer 3

''The editor is seemingly hell-bent on publishing this paper, as it has undergone 4 rounds of review, and reviewers that reject it are being replaced one-by-one.''

That's not your problem. Recommend rejection for the reasons which you give here, and then it's the decision of the editor if they still wish to publish it.

Edit: I should add, try to be tactful with your comments and don't outright accuse the authors of misconduct, at least not in such a blunt way. Even if you suspect foul play, you could be wrong.

Answer 4

You can indeed

reject this paper on the strong suspicion of foul play

Tell the editors what you suspect and why. You need not prove that what you suspect is true.

If the paper is published you can comment publicly on it in any way you like without violating reviewer anonymity. You might tell the editor that you may (or will) do that, in hopes that they will take your critique seriously.

Unfortunately, once published it may be cited forever even if debunked.

Andrew Gelman's blog has much to say on this subject.

He responded to email saying

Interesting. One thing that I didn't see in the thread is that every paper will get published somewhere, if the authors want to get it published. So getting it rejected at journal A is no big deal; it will still appear in journal B. Or maybe it could make a difference, if journal A is an attention-getter such as Jama, but otherwise not.

Answer 5

I try to never attribute to malice that which may be explainable through other mechanisms, and I don't think you need to say "I don't believe the data is real" here, especially in context of the point Trunk made.

However, you are asserting that there is a well-established literature, using solid methodology showing findings quite different from what the current authors are finding. This places an extremely high burden on the authors to convince the referees that their methodology (statistical and otherwise) is correct, and that they've done all appropriate controls.

There are always serendipitous findings. Perhaps the authors have really found something substantial about the difference between biomarkers at 3 months vs longer term, and that would merit publication.

For me, the discussion section would be key here. How have the authors tried to explain the difference between their findings and earlier studies?? Are there other controls that need to be done? Do they need to extend their findings out to 16 months to demonstrate that the biomarkers return to where the published lit would predict? Why haven't they done that??

As I said, the required level of rigor that should be used when the data doesn't match the expectations of the literature is very, very high. I can't tell you how many times I've seen authors run to the community with surprising findings, rather than trying to hunt down the artifact or confounder that really explains their results, and this may be one such case.

If the authors failed to convince you that they've employed the level of rigor required to confirm a result that disagrees with a well established literature, I'd recommend that you recommend rejection solely on that basis. I'd most certainly avoid saying "I think they just made up the data" unless you're very sure that this is the case. That's a charge of academic misconduct, which is very different from saying "I think your science sucks".

Answer 6

Medical research is not my field as I am from a physical sciences background.

But I do know a bit about statistics. The dangers of drawing conclusions from small samples are well documented. For the initiated, just run up a simple R (or Matlab) program that generates n values randomly from some normal distribution with some stated mean and standard deviation. Then let the program calculate the mean and standard deviation based on this randomly generated sample. Print the data table plus its mean and std deviation.

Now run this whole program several times each for n = 5, 10, 20, 50, 100, 200, 500, 1000, 2000, 5000, 10,000.

The results should be sobering when you compare mean and standard deviation data based estimates for small n. It can change a lot from one generated sample to another.

It might also be useful, given the OP's observation that log-normal statistics is generally regarded as the most appropriate for the phenomenon in question, that the above simulation be re-run with a log-normal distribution.

If uninitiated in statistics, please get some assistance from a colleague in the statistics function of your organization.

You might append your tables to your review comments for your editor.

Answer 7

Instead of reviewing the paper you can reject it, which seems that you should do, you could try publishing a note somewhere debunking the study, or at least indicating the issues it has. Scientific criticism and responses are not as 'in' today, but there are ways to at least leave a trace--the easiest one being ResearchGate. Another issue here is that it seems the publisher seems to have wrong incentives, which is a generator of problems such as the one you described here.