23

I recently read a paper that conducted an experiment, analyzed it, and reached a conclusion. However, the way they conducted the analysis is seriously flawed and cannot be used to support the conclusion.

As far as I can tell, the experiment is valid and only the analysis is problematic. Thus the paper can be rewritten, although the conclusion may completely change.

The paper is published in a highly-reputable and prestigious scientific journal. The authors are all senior researchers at reputable institutions. It's a bit of a surprise that this flaw got past the authors and peer review. I suspect that, because the conclusion confirms what many people already believe, the analysis was not scrutinized too closely. I only became suspicious of it because the measured effect was too strong. The analysis is also reasonably complex and the flaw is somewhat subtle.

I contacted all three authors by email and explained the problem with their analysis. I did by best to phrase the email appropriately.

A month later, I have received no response to my email. What would be a reasonable course for further action? Options include:

  1. Send the authors a follow-up email. (If so, what should I say to get the message across?)
  2. Contact the journal with my concerns.
  3. Write a response. (Would such a thing get published?)
  4. Do nothing. (I think the paper is too important.)
  • 14
    You haven’t mentioned whether you’ve asked anyone to double-check your thoughts. I’d do that first. Then academia.stackexchange.com/a/125807/11870 – Ryan Mar 2 at 15:07
  • 2
    @Ryan Yes, I got a colleague to read the paper and my email to check that I wasn’t missing something obvious. – Thomas Mar 2 at 17:18
  • 4
    Do you believe this could be a case of scientific misconduct/fraud or just a honest mistake (that collides with your data/views)? I believe each of these issues require a completely different approach. – Quora Feans Mar 2 at 18:07
  • 2
    @QuoraFeans It must be an honest mistake. – Thomas Mar 2 at 18:20
  • 1
    What is your interest in this subject? Are you a professional yourself? – Bernhard Döbler Mar 2 at 22:55
39
  1. Write a response. (Would such a thing get published?)

This. Such things are usually titled "Comment to..." and, yes, they are published, typically alongside with a reply from the authors of the commented paper (the comment is usually sent to them by the journal editor).

As usual disclaimer, since things may vary across fields and journals, check if the journal in question has already published comments of this type and, in doubt, contact the editor.

  • 1
    I suggest you to consider also the answer by guest as for a comment is indeed harder avenue. – Alchimista Mar 2 at 14:26
  • Or just publish a paper pointing out the flaw in the analysis and making clear what can and cannot be concluded from the experiment. If the original paper is in a high profile journal you might not get into the same one, but even publishing in a different journal would be worthwhile. This is really the best outcome - readers get to know the flaws in the original analysis, and you get the credit for pointing them out. – Nathaniel Mar 3 at 4:47
14

There are about a gazillion papers with problems with them. And authors who don't want to fix them, don't think they're wrong, whatever.

  1. Just resign yourself to the imperfection of the published literature. Really, killing yourself with worry that there is a science paper with a mistake in it is like the XKCD cartoon about "someone is wrong on the Internet". https://xkcd.com/386/

  2. Failing that, write a paper of your own to correct/dispute the issue. Either a direct comment/critique (harder avenue). Or a paper with some new contribution but that allows revisiting the work of the other group and dissing it en passant (easier avenue).

  • 8
    I don’t lose sleep about bogus papers at third-tier publication venues, because no one will ever read them. But this is different. – Thomas Mar 2 at 3:19
  • 6
    Science/Nature have a worse record than ACS journals in my experience. They chase a lot of hype science. – guest Mar 2 at 3:22
  • 8
    I'm not really sure comparing scientific, peer-reviewed literature being wrong to someone on the internet being wrong is the best analogy here. I think it's pretty upstanding to worry about a scientific paper being incorrect (and part of what science, in general, is all about). – AmagicalFishy Mar 2 at 16:28
3

If by analysis, you mean statistics/informatics, you could possibly do the analysis correctly and submit it as a response. The conclusion might be the same, but it should still be sorted out.

  • I do not have access to their data, which I would need to re-analyze it. – Thomas Apr 30 at 17:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.