70

I'm a sociology undegrad working on an essay for a methods class. I'm also planning on submitting it as a sample for my application to grad school. I don't want to be too specific, but I believe that this work is quite original and my hypothesis would confirm previous literature, and all in all I think it would would make a good impression on the admissions committee.

So basically I've run the tests and I'm getting conflicting results. Using one dataset (which has more observations) gives me very significant results, while using another one (which would arguably be more accurate) doesn't give me anything. So here I am at a crossroads, and I've come up with three possible options as to what to do:

  1. Only show the significant results. After all, this is just a ten-page essay, it's not supposed to be publishable or anything, right?

  2. Only use the better dataset and admit that there just isn't much there - maybe blaming it on the small sample size or on the not-so-good dependent variable. Hopefully the committee would appreciate the honesty and the relatively advanced methods that I used.

  3. Show results from both datasets, suggesting that the differences might be due to the sample size or maybe to chance.

As I type this I'm leaning more towards option 3, but I'd like to hear from people with more experience in academia. What should I do?

Improve this question
  • 85
    Contradictory results are the first step towards a discovery. – henning -- reinstate Monica Dec 10 '18 at 17:21
  • 59
    @henning ...or a debunking of scientific credos. Embrace the contradiction. – Captain Emacs Dec 10 '18 at 17:23
  • 26
    "this work is quite original and my hypothesis would confirm previous literature" It confirms existing previous results, but it's original? – Acccumulation Dec 10 '18 at 18:23
  • 7
    +1 for asking. I strongly recommend you visit Andrew Gelman;s blog regularly for discussions of the proper way to do statistics, particularly in the social sciences, Here;s one example andrewgelman.com/?s=file+drawer – Ethan Bolker Dec 10 '18 at 18:39
  • 8
    Everybody seems to agree, but then bizarrely you have so many papers published with amazing results on hand-picked datasets that nobody can reproduce on any other dataset :-) – jcaron Dec 11 '18 at 9:47
196

In research, you don't set out to prove that something is true. You set out to discover whether or not it is true. This would be knowledge. The other is just propaganda.

Negative results are not a failure. They give you evidence just as do positive results. If you ignore, or obscure, results you are lying to yourself and others. If you design an "experiment" so that it is guaranteed a priori to produce positive results, it isn't research.

Hoping that something is true isn't evidence. Many researchers start out with that idea. I think this is true. I really want it to be true. But if it is false, it is just as valuable (possibly more so) to know that and to be able to investigate why.

Report all your results. Try to explain why different aspects lead you in different directions. Only then can your learning begin.

Improve this answer
  • 34
    I really want this answer to be true, but is it... – user541686 Dec 11 '18 at 9:37
  • 7
    This, for the most part. I want to point out that there seems to be (and it's not good) this perception that somehow the only "good" results are ones that reach a novel conclusion. But this is far from true. Reaching an expected or conventional conclusion through an untested or novel path is just as much new knowledge, because it still moves that path from "predicted" to "knowledge". Moreover, even replication of an old path still has some use in that it increases confidence in those existing results, esp. if there was prior doubt about them. Replications are important. – The_Sympathizer Dec 11 '18 at 10:44
  • 3
    [And also, I might point out, verifying the predictions of an accepted theory through a novel test also serves to increase confidence. Those "confidently accepted" theories don't just get that way by magic or by fiat.] – The_Sympathizer Dec 11 '18 at 10:47
  • 3
    "You set out to discover whether or not it is true." I agree the overall goal is to determine if a hypothesis is true/false, but the way you've stated this is critically incomplete, IMO. In order to determine if a hypothesis is true we try to disprove it. We design experiments, studies, etc. with the intent to try to falsify the predictions made by the hypothesis. Only after something has withstood attempts at disproving it do we consider it to likely be true, under the conditions tested. IMO, this is a critical area that is often misunderstood, usually due to people wanting to be "right". – Makyen Dec 12 '18 at 20:00
  • 1
    @Makyen, I would state it a bit differently. A positive result in a statistical study provides evidence that the Hypothesis might be true, not proof that it is. If the study is replicated often enough, the evidence builds. But only a complete population study can provide proof, and that may be impossible if the population changes with time, making exact replication impossible. We get evidence, not proof. – Buffy Dec 12 '18 at 20:05
83

Omitting negative findings and selectively reporting only the positive findings would be a breach of research ethics. As a researcher you are supposed to uncover knowledge,* not to obscure it. Findings are often contradictory and in need of interpretation. By explaining how you obtained these contradictory results (i.e. your methods), you help others to avoid dead ends in the future and to make sense of what looks confusing today.

*Interestingly, the knowledge that research creates often takes the form of higher-level confusion rather than ultimate certainty.

Improve this answer
  • 17
    +1 because research ethics aren't something that applies only when something is "publishable" (as in "After all, this is just a ten-page essay, it's not supposed to be publishable or anything, right?") – De Novo Dec 10 '18 at 19:14
28

How honest should I be in disclosing not-so-exciting results?

You should always be completely honest: Show the results of both datasets and let the conclusion follow from the data. Comment objectively on the quality of the two datasets, and their sample sizes, but don't exclude data merely because it gives undesirable or unexciting results. In terms of the differences between the datasets, if you know why they are different then explain this, and if you don't know why they differ, then say so - don't present your speculations as scientific conclusions.

Improve this answer
  • 3
    I very much like this answer. I have a lot of respect for papers which are honest — papers which "show off" results, and obscure the honest assessment of the author's results often cost other researchers a lot of time. If things are not "as true as the author claims" a lot of time can be wasted trying to learn a technique or reproduce a result, which turns out to be not useful at all... – Earthliŋ Dec 14 '18 at 19:24
13

For option (3), add 'or there is something I do not yet understand going on".

This is much more interesting.

Your undergraduate course is there to teach you how to answer questions.

The important thing in research of any discipline is not getting the right answers but asking the right questions.

So, present both data sets, call out the discrepancy and try to explain why that is interesting and why it is worth following up.

Setting out a mini research problem like this could make you stand out much more than simply having a result.

Improve this answer
  • This does jump out as the best course. On one hand, it's crucial that proper methods are used, even if they lead you nowhere. On the other hand, if you seek publication, it has to be of interest to someone. Comparing jump heights of cat fleas and dog fleas does benefit from the conclusion that they are different. For you, the interesting bit is the difference between the datasets. Warning: you're sure to be asked about it, so you either have to delve into that a considerable bit, or explain why it falls outside the scope of your work. – kaay Dec 17 '18 at 11:33
5

I'm only a student too (graduate level), but here are a couple more reasons to go with option 3 of showing both data sets:

  • As mentioned in henning's comment, perhaps you can use your unusual results as a stepping stone for further research, and include this in your application. Treating unsatisfactory results in such a way can show that you have motivation and resilience.

  • If you did good work and showed it, even without getting "good results", that can show that you at least have potential.

  • Furthermore, in the context of applications where people usually put only their best foot forward, your honesty may actually be appreciated and respected by the admission committee. It can show that you put science first.

Improve this answer
4

Are your significant results a large effect size, or just a tiny change that is significant because of the large sample size?

Are your non-significant results similar in direction and magnitude to the significant results from the other dataset?

Consider how much the size of the dataset is impacting what you are seeing - you may be able to frame one study as confirming the results of the other if they are in agreement apart from significance. Look at more than just the p-values, especially if they are coming from a very large dataset.

Improve this answer
2

Consider for a moment that you may be comparing datasets (and results from them) incorrectly. "Significance" or rather the power is not independent of design. If Study A is done on 1,000 people but Study B is identical but includes only 100 volunteers, Study A is much more powerful, so (statistically) significant findings from A and (statistically) non-significant findings from B are non-surprising. There are better methods for comparing two studies, like a forest plot.

I only mention this because it all depends on the "you" you are trying to sell for this application. An undergraduate level sociologist doesn't need to have a graduate level statistics education, but if you are boasting it as a strength, you should be sure that you are correctly interpreting a set of findings.

The word "negative" (result or study) is an abuse of statistical terminology. There are issues of power, context, and precision; but adept researchers are readily throwing the baby out with the bathwater. Stop for a moment and think: "Do not reject H_0" means that the confidence limits include the null hypothesized value(s): 0 for differences or 1 for ratios. So what?

1) Was this study sufficiently powered or is it a complete shot-in-the-dark? Large, untenable confidence intervals can represent a crappy study or it can reflect substantial heterogeneity in the population. Were there issues with recruitment or compliance? Did you need to compensate people better? Did you administer an existing instrument and if so, did you assess yourself or the patients to be sure the wording is clear? If it's a trainwreck study you can focus on lessons learned. E.g.

we recruited 30 people based on an incorrect power calculation, our effect estimate had a much smaller magnitude than was noted in previous literature. This is a cause for some concern given our calculation was based on previous research which claimed that...

2) Is the CI narrowly on 0 or 1 excluding all other research? This is a significant finding because it is inconsistent with other literature. There's a whole field of research devoted to determining the effects of publication bias. Funnel Plots show the expected distribution of effects from meta-analyses. If the distribution is shifted with a gap at H0 it gives some pause as to whether the state of evidence is exaggerated by filtering out null findings? Important landmark research has been able to conclusively say, "No. A certain treatment does not / cannot cause a difference.

3) Is the CI wide but centered on a result which confirms previous research. For instance:

A 5,000 person study of salt reduction found that the HR for MI was 0.95 95%CI 0.92, 0.99 (p < 0.05). A confirmation study of 100 found a HR for MI of 0.95 95% CI 0.5, 1.45. (p > 0.05).

Importantly these studies agree 100%.

Improve this answer
0

That sounds so super interesting. There are statistical issues at play for sure, I don't want to dissuade you but you need to make sure that you did the math (including data collection and methodology are correct), but you can write a very powerful paper by comparing two methods. Something like:

  • Method A, which is cheap and easy to collect data on but we have concerns that it will contain bias gives a positive result.

  • On the other hand Method B that is difficult and expensive to collect data on but is far more thorough and not expected to contain bias gives a negative result.

  • Therefore, researchers should avoid using method A.

I'd be willing to bet that you could get a journal to publish a paper that is written like that, given that all of the analysis, data collection, etc. was above board, let alone get a good grade in the class.

Improve this answer
  • Actually, your "Therefore..." isn't surprising at all, so likely not a candidate for publication. Using any methodology that is expected to contain bias is flawed, unless you have ways to measure it (adding cost...) – Buffy Dec 13 '18 at 20:21
  • Nah, I think you misunderstood me because I slapped this answer together on a lunch break. That's my bad. The way that I was thinking about it was that the OP had a bunch of data, apparently collected in two different ways. – Ryan Dec 14 '18 at 22:32
  • From OP "Using one dataset (which has more observations) gives me very significant results, while using another one (which would arguably be more accurate) doesn't give me anything." It wasn't known a priori that method A was biased. I believe the paper that I was envisioning is more of a demonstration that method A is biased when that wasn't known beforehand. – Ryan Dec 14 '18 at 22:40
  • Of course you have to do a thourough literature review and make sure that the bias you are reporting isn't known already, but that's what research is about. – Ryan Dec 14 '18 at 22:41

Not the answer you're looking for? Browse other questions tagged or ask your own question.