What should be done when the theory behind a PhD thesis turns out to be wrong?

Question

I have a friend who has been working on a PhD dissertation for over 3 years and he is supposed to be wrapping it up. Unfortunately, his thesis involves a complex model and populating the model with hard-to-obtain data. After having finally gathered all the data and plugged it into the model, he is finding that the theory is more or less non-predictive and does not work. In other words, he had an interesting theory, having been tested against the data it turns out the theory is wrong.

So, in this situation what would you do? Just write a dissertation that says, "well four years of research shows that this theory is wrong"?

Answer 1

Why was the theory wrong? "It didn't work" is not the end -- the end is knowing why it didn't work. There are at least two possible answers:

• There is some fundamental, obvious mistake at the core of the model that should have been caught a long time ago. In this case, it may be a matter of starting over. It's difficult to produce a paper/thesis if the story is "I made a stupid model and it didn't work."
• There is some previously-unknown reason why the model didn't work despite smart people thinking it would. This is likely an interesting, publishable result. In fact, it could be even more interesting than a paper that just says "we used a model and it worked as expected."

It sounds like the data is valuable as well. Even if it is a matter of starting over again, this dataset sounds like it could lead to a publication or be an ingredient in the new thesis.

Answer 2

Research isn't about taking a statement and proving that it is true. That would assume that it is, in fact, true without the research being done. Research is about determining if something is true or not.

But there are two possibilities here. One, which occurs in statistical studies, is a result that shows insufficient evidence that the theory is true. Another, that can occur there or in, say mathematics, is a result that shows that the theory is clearly wrong.

The first of those is a failed experiment. It may say little about the model. But the latter is a definite result. "We thought this might be true and learned that it isn't".

Note that in statistical testing, some fraction of the samples used will give false results. If the study is well designed, then you can predict what that fraction is, say .01.

But showing that a model or a theory is clearly wrong results in a good thesis and yes, you say "This theory is wrong!!!"

A failed statistical theory may, however, require more work, or the advisor and committee may accept it, provided that the design was good and everything else done carefully and in good faith.

Knowledge is knowledge and it is good, whether positive or negative.

In fact, many published studies are later proved false because the investigators started out with a preconceived notion of the truth and did what they could to "make it so." That isn't research. Research is stepping into the unknown to try to make it known.

Answer 3

You talk to your advisor about it, then write a paper explaining your methodology and results.

Obviously, the first step is to talk to your supervisor about it, to make sure that everyone is on the same page, and to ascertain the requirements for a PhD thesis in your particular department. In some cases, you might have several peer-reviewed papers already published that you can use to compile together to create your PhD thesis; in other cases, this might not be allowed, or it might not be possible if you've yet to publish anything. Regardless of the situation, though, they should understand the requirements your department places on you to successfully complete your PhD.

However you go about structuring your thesis paper, though, you'd probably wind up explaining the methodologies you've used to produce the model, gather the data, and then analyze the results of the model using the data to make the conclusion that there's insufficient support to conclude that your hypothesis was true. Depending on the norms of your department, you might or might not publish the code you used to produce your model open-source.

Answer 4

Your question illustrates that publication bias starts in graduate school (or even earlier). To quote the Wikipedia article:

Publication bias is a type of bias that occurs in published academic research. It occurs when the outcome of an experiment or research study influences the decision whether to publish or otherwise distribute it. Publishing only results that show a significant finding disturbs the balance of findings, and inserts bias in favor of positive results.

If this research is based upon a famous preexisting model, perhaps that model has only survived for as long as it has because publication bias has shielded it from refutation. Not only is research which goes against a model adequate for a Ph.D., it could be subtly harmful to the wider research community to bury it. I obviously don't know the particulars of the case, but based on what you have said, this sounds like a good dissertation to be proud of, rather than something to be apologetic for.

Answer 5

IMHO. First thing to do is advise the thesis advisor immediately. There is an integrity question that must always be in the forefront -- the advisor must never have an excuse to question your friend's veracity. This is important for the 2nd step. If the advisor personally supports the discovery of something that is not true, he will talk to his peers and feel out support for a formal defense. If he has support from his peers, he'll let the defense go ahead (with a rewrite, of course). After all, isn't this the purpose of a thesis -- to discover something that wasn't known before? Remember, the advisor also has "skin in the game". If he let's a defense proceed and it's shot down -- he loses "face". If he believes in your friend, he/she have a good shot.

The people, their relationships and the organizational aspects of the process are usually more important than the final output, but there is no fundamental reason why a negative result isn't still discovering something worth reporting. After all, the drug Sildenafil failed its clinical trials until somebody read a interdepartmental memo and looked at the negative-results differently.

OBTW. I spent 20 years at a Fortune 500 company, a chunk of that time as a manager. I don't have a PhD, but I do know people and processes. Your friend's problem is no different. Tell your friend that he has this and congratulations are in order for finishing the research! Encourage him not to be a ABD -- your friend is so close!!!

Answer 6

If you knew the answer before beginning, it wouldn't really be research.

There've been plenty of theories that turned out to be wrong. A recent example is supersymmetry. This particle physics idea was really attractive for many reasons, and many physicists spent their careers working on the theory. Still, the LHC hasn't detected supersymmetric particles, which is proving to be a major problem.

Your friend should talk to their advisor about the issue, but should not be too concerned about turning in a thesis that says "well four years of research shows that this theory is wrong". He'd be in good company.

Answer 7

My advisor have told at the beginning of the workshops, that the thesis that turned to be wrong characterize usually the best dissertations.

The explanation is simple: if your thesis proves to be correct, than you've actually, in most cases, proves something that is believed to be correct to be correct. In most cases, it doesn't move anything forward. On the other hand, if your thesis was proved to be wrong, that's much more interesting. You've surely discussed your initial thesis with your adivisor, it also matches your initial state of knowledge you've got from books. But then you've found something that doesn't match that state of knowledge. If you can find out why your original thesis was wrong, it has a big potential to bring something new to the science.

Answer 8

Working with what you have:

• Hard to obtain data
• A theory that you now know for sure is wrong

Wrong in what way? "Can't show statistical significance" or "can show the opposite is true"? If you can say something definite such as that the opposite is true, then you can formulate a new model that's better than your old one (because it agrees with the evidence).

If all you have is "no significance" then that's less juicy, but you may still be able to write a paper on "model X fails in circumstance Y, and here are my arguments why I think Z is the cause".

The hard to obtain data are an interesting prospect. If it took years to get this data, then publishing it so that someone else doesn't also need years to collect those data can be scientifically valuable. If it took years to get this data, presumably other people haven't had much chance to study and analyze the data, and you can also write about that.

Altogether, you have stuff to write about that took a smart person three years of serious work to gather. And you've learned about the ups and downs of research. Work with your supervisor to figure out a new direction for the thesis. But you should be able to write a strong thesis that demonstrates that you've learned how to do research, and presents outcomes of research that weren't already available to the scientific community (this hard to obtain data and knowledge about which modeling approach surprisingly doesn't work).

Answer 9

I would structure a document entitled "On the non sustainability of the hypothesis that ...".

This document could be a research report, a journal article, a thesis chapter, anything. While writing, the logic of the thoughts will unfold itself; perhaps you can even find that it is not true that all is wrong. All other reactions given in this thread contribute to providing motivation and sensible ways forward.

In terms of self-reflection, beside a bunch of frequently occurring biases of judgement, please mind the impostor's bias as well: there is even a playlist of TED talks to choose the more suited variant from.

Finally, for inspiration, it could be interesting to look at the editorial lines of journals that publish 'studies of failures' as different from 'failed studies'. One in the medical community that I heard of is the Journal of Negative Results in Biomedicine. I imagine it is important to gather up the vocabulary appropriate to explaining 'studies of failures', and borrowing from any reputable source can be useful.