69

I have a friend who has been working on a PhD dissertation for over 3 years and he is supposed to be wrapping it up. Unfortunately, his thesis involves a complex model and populating the model with hard-to-obtain data. After having finally gathered all the data and plugged it into the model, he is finding that the theory is more or less non-predictive and does not work. In other words, he had an interesting theory, having been tested against the data it turns out the theory is wrong.

So, in this situation what would you do? Just write a dissertation that says, "well four years of research shows that this theory is wrong"?

  • 1
    Is this their own model or an already published one? – Bryan Krause Dec 6 '19 at 0:22
  • 1
    @BryanKrause It is a model derived from a pre-existing famous model. – Tyler Durden Dec 6 '19 at 1:00
  • 99
    @TylerDurden Showing a model doesn't work, or fails in a certain case, can be way more interesting than showing a known model works. – Bryan Krause Dec 6 '19 at 1:49
  • 1
    Related, not quite a duplicate: academia.stackexchange.com/questions/30995/… – Ethan Bolker Dec 6 '19 at 3:13
  • 2
    Your PhD shows that you can do science the right way. Your title is PhD, not PhD in The Effect Of Radiation On Multiple Bodies In A Vacuum In The Context Of Spherical Cows. Having a thesis which proves that something is wrong somehow sucks, but it is not a problem per se with the fact of getting the title of doctor. – WoJ Dec 7 '19 at 13:53
131

Why was the theory wrong? "It didn't work" is not the end -- the end is knowing why it didn't work. There are at least two possible answers:

  • There is some fundamental, obvious mistake at the core of the model that should have been caught a long time ago. In this case, it may be a matter of starting over. It's difficult to produce a paper/thesis if the story is "I made a stupid model and it didn't work."
  • There is some previously-unknown reason why the model didn't work despite smart people thinking it would. This is likely an interesting, publishable result. In fact, it could be even more interesting than a paper that just says "we used a model and it worked as expected."

It sounds like the data is valuable as well. Even if it is a matter of starting over again, this dataset sounds like it could lead to a publication or be an ingredient in the new thesis.

  • 33
    The practical problem, in my experience, is that almost all problems look obvious in hindsight :) that's a rarely discussed reason to be able to point to previous literature about your theory, or makes it harder for a reviewer to say "well, every expert knew this wouldn't work". – xLeitix Dec 6 '19 at 9:15
  • 2
    It's the difference between watching someone doing surgery vs doing one yourself. – Nelson Dec 7 '19 at 13:49
44

Research isn't about taking a statement and proving that it is true. That would assume that it is, in fact, true without the research being done. Research is about determining if something is true or not.

But there are two possibilities here. One, which occurs in statistical studies, is a result that shows insufficient evidence that the theory is true. Another, that can occur there or in, say mathematics, is a result that shows that the theory is clearly wrong.

The first of those is a failed experiment. It may say little about the model. But the latter is a definite result. "We thought this might be true and learned that it isn't".

Note that in statistical testing, some fraction of the samples used will give false results. If the study is well designed, then you can predict what that fraction is, say .01.

But showing that a model or a theory is clearly wrong results in a good thesis and yes, you say "This theory is wrong!!!"

A failed statistical theory may, however, require more work, or the advisor and committee may accept it, provided that the design was good and everything else done carefully and in good faith.

Knowledge is knowledge and it is good, whether positive or negative.

In fact, many published studies are later proved false because the investigators started out with a preconceived notion of the truth and did what they could to "make it so." That isn't research. Research is stepping into the unknown to try to make it known.

  • 25
    "You don't use science to show that you're right, you use science to become right." - Randall Munroe – user2390246 Dec 6 '19 at 15:52
  • A lot of research is definitely taking a statement - call it "a conjecture" if you will - and proving that it is true. Perhaps not in the more applied fields, but in theoretical ones - definitely. – einpoklum Dec 7 '19 at 20:05
  • 1
    @einpoklum-reinstateMonica, but the conjecture isn't known to be true at the start. It may even be false. Research is gathering evidence for one or the other. Definitive evidence is best. But the attitude should be "Is this true?", not "This is true." at the beginning. Too many researchers get committed to the truth of their hypothesis and wind up getting into various errors. Even in math and theoretical physics, the statement comes first, but you need the proof. – Buffy Dec 7 '19 at 20:10
  • 1
    In math a counterexample to some conjecture is important, there are dozens of examples out there. And "impossibility" proofs are of great value after identifying why something is not working, a famous example would be the negative solution to the Entscheidungsproblem and Gödel's incompleteness theorems. – StefanH Dec 8 '19 at 0:04
  • In theoretical fields, "Known to be true" = "Proven" (essentially). – einpoklum Dec 8 '19 at 7:16
18

You talk to your advisor about it, then write a paper explaining your methodology and results.

Obviously, the first step is to talk to your supervisor about it, to make sure that everyone is on the same page, and to ascertain the requirements for a PhD thesis in your particular department. In some cases, you might have several peer-reviewed papers already published that you can use to compile together to create your PhD thesis; in other cases, this might not be allowed, or it might not be possible if you've yet to publish anything. Regardless of the situation, though, they should understand the requirements your department places on you to successfully complete your PhD.

However you go about structuring your thesis paper, though, you'd probably wind up explaining the methodologies you've used to produce the model, gather the data, and then analyze the results of the model using the data to make the conclusion that there's insufficient support to conclude that your hypothesis was true. Depending on the norms of your department, you might or might not publish the code you used to produce your model open-source.

  • I wouldn’t say “obviously...”, otherwise the OP wouldn’t have asked the question. – Andy Clifton Dec 9 '19 at 7:34
15

Your question illustrates that publication bias starts in graduate school (or even earlier). To quote the Wikipedia article:

Publication bias is a type of bias that occurs in published academic research. It occurs when the outcome of an experiment or research study influences the decision whether to publish or otherwise distribute it. Publishing only results that show a significant finding disturbs the balance of findings, and inserts bias in favor of positive results.

If this research is based upon a famous preexisting model, perhaps that model has only survived for as long as it has because publication bias has shielded it from refutation. Not only is research which goes against a model adequate for a Ph.D., it could be subtly harmful to the wider research community to bury it. I obviously don't know the particulars of the case, but based on what you have said, this sounds like a good dissertation to be proud of, rather than something to be apologetic for.

7

IMHO. First thing to do is advise the thesis advisor immediately. There is an integrity question that must always be in the forefront -- the advisor must never have an excuse to question your friend's veracity. This is important for the 2nd step. If the advisor personally supports the discovery of something that is not true, he will talk to his peers and feel out support for a formal defense. If he has support from his peers, he'll let the defense go ahead (with a rewrite, of course). After all, isn't this the purpose of a thesis -- to discover something that wasn't known before? Remember, the advisor also has "skin in the game". If he let's a defense proceed and it's shot down -- he loses "face". If he believes in your friend, he/she have a good shot.

The people, their relationships and the organizational aspects of the process are usually more important than the final output, but there is no fundamental reason why a negative result isn't still discovering something worth reporting. After all, the drug Sildenafil failed its clinical trials until somebody read a interdepartmental memo and looked at the negative-results differently.

OBTW. I spent 20 years at a Fortune 500 company, a chunk of that time as a manager. I don't have a PhD, but I do know people and processes. Your friend's problem is no different. Tell your friend that he has this and congratulations are in order for finishing the research! Encourage him not to be a ABD -- your friend is so close!!!

5

If you knew the answer before beginning, it wouldn't really be research.

There've been plenty of theories that turned out to be wrong. A recent example is supersymmetry. This particle physics idea was really attractive for many reasons, and many physicists spent their careers working on the theory. Still, the LHC hasn't detected supersymmetric particles, which is proving to be a major problem.

Your friend should talk to their advisor about the issue, but should not be too concerned about turning in a thesis that says "well four years of research shows that this theory is wrong". He'd be in good company.

  • 9
    Really, the fact that you may not get a PhD (or may write an extremely weak thesis and be in a shitty position to move your academic career forward) is not something to lose sleep over? Can you clarify what you mean exactly? These vague metaphors about losing sleep and being in good company aren’t really answering the question. – Dan Romik Dec 6 '19 at 3:10
  • The supersymmetry part may be controversial :-) It's true that most SUSY models predicted sparticles that would have been discovered by now. But that just encourages more creative models (e.g., "stringy naturalness") – cag51 Dec 6 '19 at 3:12
  • 6
    @Dan Romik: In many fields/institutions/cultures the quality of the thesis is not measured by whether the investigated theory comes out positive. So the problem explained here will not result in not getting a PhD or having a very weak thesis (although the thesis of course can be weak for other reasons, so it's always good advice to worry and lose sleep;-). This however depends on field/institution/culture, so discussing this with the supervisor is mandatory. – Lewian Dec 6 '19 at 14:59
  • 6
    Personally as supervisor I had PhD students who wouldn't for the life of them admit that the idea that they have been investigating doesn't work that well. Sometimes it was my idea and they even thought they owe it to me to show that the idea is good, but occasionally it wasn't. Surely I am fine with a well written well founded and understood thesis with a (largely) negative result. Of course one needs to elaborate what the field can learn from this. But it can drive me to dispair that students just can't get themselves to admitting that something doesn't work well. – Lewian Dec 6 '19 at 15:02
5

My advisor have told at the beginning of the workshops, that the thesis that turned to be wrong characterize usually the best dissertations.

The explanation is simple: if your thesis proves to be correct, than you've actually, in most cases, proves something that is believed to be correct to be correct. In most cases, it doesn't move anything forward. On the other hand, if your thesis was proved to be wrong, that's much more interesting. You've surely discussed your initial thesis with your adivisor, it also matches your initial state of knowledge you've got from books. But then you've found something that doesn't match that state of knowledge. If you can find out why your original thesis was wrong, it has a big potential to bring something new to the science.

4

Working with what you have:

  • Hard to obtain data
  • A theory that you now know for sure is wrong

Wrong in what way? "Can't show statistical significance" or "can show the opposite is true"? If you can say something definite such as that the opposite is true, then you can formulate a new model that's better than your old one (because it agrees with the evidence).

If all you have is "no significance" then that's less juicy, but you may still be able to write a paper on "model X fails in circumstance Y, and here are my arguments why I think Z is the cause".

The hard to obtain data are an interesting prospect. If it took years to get this data, then publishing it so that someone else doesn't also need years to collect those data can be scientifically valuable. If it took years to get this data, presumably other people haven't had much chance to study and analyze the data, and you can also write about that.

Altogether, you have stuff to write about that took a smart person three years of serious work to gather. And you've learned about the ups and downs of research. Work with your supervisor to figure out a new direction for the thesis. But you should be able to write a strong thesis that demonstrates that you've learned how to do research, and presents outcomes of research that weren't already available to the scientific community (this hard to obtain data and knowledge about which modeling approach surprisingly doesn't work).

  • 1
    I was looking for this answer, +1. The hard-to-obtain data is hard to obtain! It can certainly be analyzed in depth to draw some conclusions, or at least confirm known ones. It will not be a thesis as strong as the one the student anticipated, but if gathering the data was hard maybe that can be the focus, instead of the result he was hoping to get from the complex model. – AnalysisStudent0414 Dec 7 '19 at 13:50
1

I would structure a document entitled "On the non sustainability of the hypothesis that ...".

This document could be a research report, a journal article, a thesis chapter, anything. While writing, the logic of the thoughts will unfold itself; perhaps you can even find that it is not true that all is wrong. All other reactions given in this thread contribute to providing motivation and sensible ways forward.

In terms of self-reflection, beside a bunch of frequently occurring biases of judgement, please mind the impostor's bias as well: there is even a playlist of TED talks to choose the more suited variant from.

Finally, for inspiration, it could be interesting to look at the editorial lines of journals that publish 'studies of failures' as different from 'failed studies'. One in the medical community that I heard of is the Journal of Negative Results in Biomedicine. I imagine it is important to gather up the vocabulary appropriate to explaining 'studies of failures', and borrowing from any reputable source can be useful.

Not the answer you're looking for? Browse other questions tagged or ask your own question.