37

Recently, I came across a PhD thesis and an article derived from that PhD thesis and found a serious flaw in them that actually makes their conclusions invalid.

In the first place, I was unsure if I'm right or not, so I contacted the guy that wrote the PhD thesis and subsequent article and described my concerns and I showed my calculations to him and asked if he thinks in fact there is a major problem in his PhD thesis and that article, which is derived from it. He responded back that embarrassingly, in fact I'm right and there is a problem but he can't do anything about it.

I didn't want to force him to do something but I'm just thinking maybe there might be a possibility to publish an erratum for at least that article. It's a major flaw and unfortunately it just invalidates the major conclusions of paper and PhD thesis. Surprisingly, the article is cited for 15 times and nobody found that obvious problem. My question: What's the best way to deal with these kind of situations that even author himself/herself admits that in fact there is a major flaw in his/her thesis or article?

  • 6
    "I'm right and there is a problem but he can't do anything about it." Why not?? – Pete L. Clark Aug 14 at 19:04
  • 8
    @PeteL.Clark That's his answer not mine that he can't do anything about it, but of course, I'm not convinced with this answer. – Alone Programmer Aug 14 at 19:05
  • 3
    Right, I agree with you. I am struggling to think of a situation in which it is not the case that he not only CAN do something about it but is ethically obligated to. What I was asking was: what reasons did he give? – Pete L. Clark Aug 14 at 19:08
  • 1
    @PeteL.Clark "what reasons did he give?" Nothing! I mean I just didn't ask why after he admitted that there is a problem and he can't do anything about it, cause I didn't want to make this impression that I found something in his work and now I want to put him in trouble. – Alone Programmer Aug 14 at 19:11
  • 2
    Can't the guy just add an erratum? – Tom Aug 16 at 12:06
57

One possibility is for you to write a paper on your reanalysis. Often this would need to be something more than pointing out the flaw alone, but that may be discipline-dependent. If the flaw you have detected is a key part in your own analysis of your work, including your updated analysis in a larger article (while citing their paper) is a smooth way to point out how you had to change the method.

The second possibility is that many journals (at least in my area) allow for Comments on articles they have published. These are for just this purpose - to point out something 'wrong' about a paper that has been published. If the editors decide to proceed with it, the original authors are usually given an opportunity to craft a Reply piece to agree, disagree, alter, whatnot. Then they are published back-to-back in the journal.

  • 1
    Unfortunately, there is no option for putting comment about articles that are published in that journal. My minor concern about your first option is that, I don't want my short letter, to show the flaw based on reanalysis, will be considered an offence to the guy who wrote the paper and PhD thesis. Despite the fact that indeed there is a major flaw in this article, he and his co-authors are big draws in my research field and I don't want to struggle with them. The main author is a nice guy, and he quickly took the responsibility for the problem, but I don't know about his other co-authors. – Alone Programmer Aug 14 at 18:01
  • 9
    @AloneProgrammer - is the (faulty) analysis in the paper something central to your work? If so, in one of your papers you could show your analysis, and state it is different from the other paper. Leave it up to readers to figure out that the other is in error (at least for your usage of it). – Jon Custer Aug 14 at 18:09
  • 1
    Yes, in fact it is related to major part of my research. This idea sounds great I believe. Thanks! – Alone Programmer Aug 14 at 18:13
  • 13
    To avoid problems with the authors of the original paper, you can invite them to write the re-analysis paper together and co-author it. – Phlya Aug 15 at 14:07
  • 8
    Knowing a bit more about the OP's work from past questions, this answer is definitely correct. This is not an area where things are truly wrong but this sounds like a flaw in approach. The correct thing to do is to publish a paper using a different approach referencing the previous paper, explain why the new approach is superior (let others decide if that makes the previous work inferior), and explain differences in the results. Or, as suggested, publish a Comment addressing the methodological errors. Do not write to the original editor, do not demand retraction. – Bryan Krause Aug 15 at 15:41
28

Do not overthink it.

I have had similar experience and authors were reluctant at first (mostly because they have moved to another problem).

It is very dangerous to leave a flaw in the literature. Especially a critical flaw where others may build on it. This will lead to more chaos. I am aware some (well-known) people do not care about their old results being incorrect. But this is not what Academia is about. You should appreciate the peer review and rigorous findings more than anything else. You already have contacted the author, offer him a collaboration (if seems right) on correcting the whole thing in the context of your work.

You usually have spotted the error because your work is related. On that particular connection re-analyze the claim and correct it. Include in your new paper the claim, prove why it is incorrect and prove the new correct result. This actually can be seen as a new contribution to your paper/thesis.

For me, I was very satisfied with the outcome; the first author of the other paper did mentor me for a while after we met at a conference.

  • 5
    I don't buy this. Descartes' whole physiological framework is just about entirely incorrect. The normal process of science is correcting it. It didn't cause chaos -- it forms the hypothesis-driven framework of the scientific method. – Scott Seidman Aug 15 at 13:32
  • 2
    @ScottSeidman errors happen indeed, and it is the author duty to make sure any result is correct before claiming it. It is also others' duty if spotted an error to fix it. But leaving a flaw in the literature just to avoid ''awkward situation'' (as suggested by the OP comment to Jon's answer) isn't the right thing. – seteropere Aug 15 at 13:48
8

Write to the editor of the journal. Offer your explanation and ask for the best way forward. You might be lucky and be given a chance to write an article with your clarification. The editor might instead decide to do a correction with varying levels of acknowledgement of you.

article is cited for 15 times and nobody found that obvious problem

The problem is not obvious despite your initial impression. It may be worthwhile dissecting and contemplating the complexity of the blindspot in this area. Was it obvious to you because you come from a different perspective? Have experience from a different discipline? Interrogate why the issue was not obvious to at least 15 publication worthy authors and not to mention a larger number of editors and peer reviewers.

  • Yes, I agree it will not be obvious unless you dig deep into the problem. I believe it was not discovered earlier by author himself or reviewers of the journal, because of huge ambiguity that exists in the literature of this area of research. It's not that difficult to catch the flaw in this paper, cause it's just about wrong Reynolds number, which could be calculated easily even by high school students. But, I'm just wondering why nobody did that before and why nobody realized simply if your Reynolds number is off by an order of magnitude, your results are simply wrong... – Alone Programmer Aug 15 at 13:04
  • Nice idea to think about why it might be that no one has reported the problem until now. Worth considering that maybe some people have spotted it but not done anything about it. Another factor to consider is that those 15 citations may not directly build on the work, they may be fairly incidental citations along the lines of "Other methods previously used in this area are X (1), Y (2) and Z (3)". – user2390246 Aug 15 at 13:05
  • Investigate and describe the ambiguity in the literature might be useful. It could be that most people don't appreciate or realise the ambiguity? It may be worth discussing with colleagues or researching whether this is a Reynolds number issue or whether it fits into a recognised category of errors? – Poidah Aug 15 at 13:19
  • @user2390246 The citations come from a close circle of researchers, who are really well known in my field, but they just cite each other. The worst part is that the result of this research come from a conference challenge in 2011 that six different research groups did an identical series of simulations and their results across these six groups are compared together. But simply, all of those simulations are built based on a wrong input (the simulations are highly sensitive to this input), which means their conclusions are wrong. – Alone Programmer Aug 15 at 13:20
  • 1
    Maybe publish a series of articles using the "right" input then. It is more useful the describe and derive concepts around why and how this sort of errors and assumptions occur. In computing world, conceptualisation work is not as appreciated as it is from the other sciences especially psychology and safety factor research. But if you can conceptualise it, then you can then use that technique or knowledge to identify other similar errors or produce new checks and balances for such discussions in the future... – Poidah Aug 15 at 13:25
8

This is what science is. Not everything that gets published is correct or true. Presumably, the literature of science is self correcting. Someone will eventually address and correct it. For example, Descartes' whole physiological framework is just about entirely incorrect. The process of science fixed this, over hundreds of years, and still Descartes publications were an important part of that process.

If the area of research you're referencing is important, the error you hold to be present will misguide the community and harm future research efforts, and you're a participant in the community with sufficient credibility and standing to correct the issue, you might consider a letter to the editor or a rapid publication to a high profile journal.

If any of those criteria do not hold, your actions should be tempered. If the publication you believe to be incorrect is on the topic of your personal research, the issue might find its way into the discussion section of one of your near-future publications. If it doesn't reach that level in your personal research portfolio, you might just let it go, and let someone more appropriate address it.

  • The research is about simulating blood flow in neurovascular network, which I think it is important... – Alone Programmer Aug 15 at 13:38
  • 3
    @AloneProgrammer "Importance" is not what is relevant here. Everyone thinks their work is important. You are not in a field where this type of error is treated as something to be removed from history. Build on the work with your better work. – Bryan Krause Aug 15 at 15:42
  • "Important" is relative. I'm talking more in terms of an error in a paper that can potentially change public health policy in an incorrect direction. Think "vaccines". – Scott Seidman Aug 15 at 15:45
  • @BryanKrause You are right. But, this one is not about our interpretation of results or even comparison of obtained results with clinical data, which I would say the variance in this area is huge and even conflicting results are not considered wrong, and as you pointed out correctly, it's up to readers and researchers to decide. But, the problem here is something more fundamental about the inputs of mathematical model, that is used essentially to build the computational tool and extract the results in this paper. A factor of magnitude lower Reynolds number is really a big concern in this area. – Alone Programmer Aug 15 at 15:56
  • 4
    @AloneProgrammer That's fine, publish your work showing something different with different model inputs. This happens all the time. A correct reading of any biological/medical research is "given these methods, we got these results" - there are thousands of papers that use methods that have been replaced with better approaches. If your new results impact the field that is fine, that means you have potentially high impact work. – Bryan Krause Aug 15 at 16:00
6

I have been in a similar situation, but on the other side. My own MS thesis and an article derived from it contained a major flaw. In short, the thesis was about investigating some static waves in a certain system, but the system itself is unstable in the very same model, with the instability length being comparable to the characteristic wavelength of the static waves in question. The whole investigation did not make any scientific sense at all, because the assumed system could not be physically realized in the first place.

I myself discovered the flaw a few months before submitting my thesis. I discovered the flaw pretty accidentally: I wanted to formally prove that the system is stable, but the result of my calculations showed that the opposite is true. I had never heard from my supervisor, who had given me the problem for my thesis, or from his colleagues that the system may be unstable. Everyone simply did not even think about that possibility.

And then I faced the dilemma as to what to do.

My final choice was to tell no one and to simply get my degree. I was an undergraduate student graduating from a Russian university, and getting my MS degree asap and moving abroad to get a Western PhD degree was my highest priority. If I had raised the issue about the flaw, I would have suffered a major setback in my career. I would have had to start my MS project over, and I am not even sure whether I would have been allowed to do it at all. If I had had a setback in my career, I would not have been able to win a prestigious PhD stipend in a Western country.

I should have checked at the very beginning of my MS project whether the system is stable. I should have, but I did not. Partially it is a fault of my supervisor, who directed my work in a very rigid way, giving me concrete tasks and deadlines. He never told me to check whether the system is stable. The official research plan, which I and he signed, did not contain any mention of a stability analysis. It was my own initiative to try to prove stability, because I felt that this was needed to make my investigation complete. I did not even talk to my supervisor about my idea to perform a stability analysis. After I discovered the flaw, I was sure that if I talked to my supervisor about the flaw, he would say the whole MS project had to be canceled.

After I submitted my thesis, my supervisor insisted that I write and publish an article derived from the thesis. I did not want to do it, but I had to. After all, I needed good recommendation letters from my supervisor, so I had to obey. The article was published in a reputable American journal and was cited ~20 times. Writing that article was the most unpleasant experience in my scientific career.

No one noticed the flaw, so I successfully got my MS degree in Russia, moved abroad, got a Western PhD degree, and some years later published an article explaining the flaw. I explicitly wrote in that article that my previous article and a number of other articles, which I cited, are invalid science. No one published a comment in response. In private conversations, my colleagues confirmed that my conclusion about the flaw is correct.

Why did I wait about six years to tell the scientific community about the flaw? Don't hate the player, hate the game. I simply could not afford a career setback. I needed my MS degree asap. I believe I made a wise choice. I would do exactly the same in the same circumstances. Who wouldn't?

Concerning your question, the principal thing is this: what's in it for you? Does the guy's article make any harm to you or your reputation by, e.g., contradicting your own articles? If it does not, then I want to tell this: there are so many wrong or misleading articles in science, there are so many awful things in science, and the guy's article is a drop in the ocean of all this. Why wouldn't you focus on doing your own great research instead? I guess the guy's article is insignificant anyway and is not even worth considering, just like most articles in science. I guess it is just a mathematical or computational exercise made for the purpose of producing a paper and earning a PhD degree. If my guess is correct, then you do not need to stoop to that level and search for mistakes in insignificant articles, and you do not need to harm the guy's career at its very beginning. I have articles published in Physical Review Letters, even as the first author, but I started my career with a wrong and misleading MS thesis.

Imagine you noticed a mistake in an article published in Indian Journal of Physics, whose impact factor is below 1. Would you care to correct the mistake by publishing a comment? I doubt you would. Not every mistake must be corrected. Some are just not worth spending time and effort correcting them.

My own principles are simple:

  1. All physics journals that are "lower" than Physical Review journals are trash bins.

  2. If an article does not interfere with my own research, then this article is none of my business.

I hope that my post will help you and others look at the issue from an angle different from the one from which many people see the issue.

  • 1
    @AloneProgrammer Of course, it is a research misconduct. But if I had not elected to cover the flaw, I would most likely not have become a scientist at all. I could not afford two more undergraduate years in Russia. My parents did not want to help me financially any further. I had to get my MS degree asap and move abroad. At that time (late 1990s), living in Russia was very hard because of an economic crisis. – Sandra Aug 17 at 8:24
  • 1
    @AloneProgrammer Technically speaking, my MS thesis and the article derived from it did not contain any mistake, because I took a model given to me and investigated static waves within the framework of that model. The fact that the model is faulty is a separate issue. In essence, the message of my thesis and the article was, "If we take that model, we get these results." I was just an undergraduate student who had to do what the supervisor said. He gave me the model and ordered to do certain calculations. I did them, and I did them absolutely accurately. – Sandra Aug 17 at 8:33
  • 1
    @AloneProgrammer As soon as I felt it was safe for my career to tell the community that the model is faulty, I did so. I know I violated the ethical norms of research conduct, but I was FORCED by the circumstances. At any rate, my MS project was insignificant research anyway. It was rather a training project that allowed me to learn how to do calculations and how to write a thesis and a research article. Even if the model were not faulty, the article would not have had any considerable impact. No one used the results of that project. People simply cited my paper. – Sandra Aug 17 at 8:41
  • 1
    @AloneProgrammer The only harm due to me covering the flaw is that a number of other scientists continued using the same faulty model for other purposes in their research, unsuspecting that the model is faulty. If I had told them that the model is faulty, they would have spent their research efforts for something more useful. But that would have put my own career in danger. Unfortunately, sometimes we face hard choices. – Sandra Aug 17 at 8:48
  • 2
    +1 for the honesty – rhombidodecahedron Aug 20 at 18:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.