164

The following might be a slight generalization for all fields but something I've noticed especially in the field of Scientific Computing:

  1. Why don't people publish failures? I mean, if they tried some experiment and realized at the end that they tried everything and nothing worked. Why don't they publish this? Is it because such content won't get published or is it because it is shameful to have a failed experiment in a journal alongside prize-winning papers?

  2. I spent a better part of a year working on, what now looks like, a dead problem. However, most papers that I read initially took you to the point of feeling optimistic. Now that I re-read the papers, I realize that I can say (with much confidence) that the author is hiding something. For instance, one of the authors who was comparing two systems, gave an excellent theoretical foundation but when he tried to validate the theory with experiments, there were horrible discrepancies in the experiments (which I now realize). If the theory wasn't satisfied by the experiments, why not publish that (clearly pointing out parts of the theory which worked and which didn't) and save the future researchers some time? If not in a journal, why not ArXiv or their own websites?

Improve this question
9
  • 39
    that is one of the points of open science. You publish failed approaches on your blog or wiki. I remember reading sometime ago, that failed studies (i.e. drug not doing anything) being unpublished was becoming a problem in certain parts of medicine research and now big NIH studies (I think) have to be registered in a database before they are conduced and include their results regardless of if they are positive or not. I can't remember a source for this unfortunately :(.
    – Artem Kaznatcheev
    Commented Mar 15, 2012 at 7:03
  • 2
    Nice list Joel Reyes Noche. However, among the 6 journals that you have mentioned, the top five are focused on papers with negative results. The last one, the Journal of Errology is focused towards providing a medium for those unpublished futile attempts behind every published paper that are usually discarded. These are way more common that the other papers, for which most researchers don't have time to publish, since they don't result in any benefit for their careers.
    – user451
    Commented Mar 16, 2012 at 11:40
  • 4
    I'm posting the rest of Selvin's comment here (formerly submitted as an answer, but converted because it doesn't answer the question): "This reminds me on Edison's practices, where he kept a list of all the things that did not work, which later helped him in a major way. Imagine doing 10,000 different experiments, without properly keeping track of each experiment. It is going to be a nightmare."
    – aeismail
    Commented Mar 17, 2012 at 22:31
  • My dissertation research was a null result. The article on it has more than 80 citations. But that is in particle physics where setting limits on the onset of an expected phenomena is a very common thing.
    – dmckee --- ex-moderator kitten
    Commented Oct 9, 2013 at 5:19
  • 1
    (I posted this as answer but got removed as it is not really an answer. So, posting as a comment) Partially inspired by this discussion and partially by my own experience, we, a bunch of researchers in computational science are organizing a workshop titled "E-science ReseaRch leading tO negative Results (ERROR)". The workshop will be held in conjunction with the eScience 2015 conference in Munich, Germany. For more information about the workshop see press3.mcs.anl.gov/errorworkshop.
    – Ketan
    Commented Mar 11, 2015 at 16:55

8 Answers 8

Reset to default
93

"Why don't people publish failures?"

Actually, they do.

  1. Journal of Negative Results (ecology and evolutionary biology)
  2. Journal of Negative Results in Biomedicine
  3. Journal of Pharmaceutical Negative Results
  4. Journal of Interesting Negative Results (natural language processing and machine learning)
  5. Journal of Negative Results in Environmental Science (no issues yet?)
  6. Journal of Errology (no issues yet?)

and so on...

(You might also want to see the Negative Results section of the Journal of Cerebral Blood Flow & Metabolism.)

Improve this answer
7
  • 26
    they do, but not often, and not in 'standard journals'
    – David LeBauer
    Commented Mar 19, 2012 at 4:56
  • 35
    Few to none of these journals have received many submissions. This is tragic because the lack of published replication attempts is one of the biggest problems in science (see Ioannidis classic paper, "Why most published research findings are false"). Reforms are needed to create greater incentives for scientists to do and publish such studies. We are approaching publishers about a new journal concept in which only the study plan (methods and analysis plan) for a replication attempt would be reviewed (plus.google.com/113040210411045341720/posts/BB2BnpHi8Tw), then publication guaranteed.
    – Alex Holcombe
    Commented Apr 5, 2012 at 10:51
  • 6
    In undergrad, we fantasized about creating the "American Journal of Failures and Missteps".
    – geometrian
    Commented Jan 15, 2015 at 2:36
  • 1
    I'd also like to add that debunking an established theoretical idea is very worthy of publication, even in a journal that publishes positive results. And it does happen (unfortunately not often enough).
    – WetlabStudent
    Commented Mar 11, 2015 at 23:55
  • 1
    There's also a practical reason: writing a good paper (even if on negative) results takes time and effort which is something most researchers prefer to spend otherwise -- for instance on achieving positive results (perhaps in the hope to publish those at more prestigious venues).
    – Thomas
    Commented Nov 24, 2016 at 8:40
48

Null results are hard to publish. They just are. Interestingly enough however, in my field they are not the hardest thing to publish. The general order goes:

Well powered (big) studies that find what people expect
Poorly powered (small) studies that find what people expect
Poorly powered studies that find the opposite of what people expect or null findings
Well powered studies that find the opposite of what people expect

Those middle two categories are where you'll find most "failures", at least in terms of finding a statistically meaningful effect. That being said, there's an increasing push to see these types of studies published, because they're an important part of the literature, and several medical journals have made fairly remarkable steps in that direction - for example, if they accept a paper on the protocol for an upcoming clinical trial, they also commit to publishing the results of the trial (if they pass peer review) regardless of the finding.

When it comes down to it, I think there's three reasons negative results aren't published more beyond "it's hard":

  1. Lack of pay off. It takes time and thought to get a paper into the literature, and effort. And money, by way of time and effort. Most null findings/failures are dead ends - they're not going to be used for new grant proposals, they're not going to be where you make your name. The best you can hope for is they get cited a few times in commentaries or meta-analysis papers. So, in a universe of finite time, why would you chase those results more?
  2. Lack of polish. Just finding the result is a middle-step in publishing results, not the "and thus it appears in a journal" step. Often, its easy to tell when something isn't shaping up to be successful well before its ready for publication - those projects tend to get abandoned. So while there are "failed" results, they're not publication ready results, even if we cared about failures.
  3. Many failures are methodological. This study design can't really get at the question you want to ask. Your data isn't good enough. This whole line of reasoning is flawed. Its really hard to spin that into a paper.

Successful papers can be published on their own success - that is interesting. Failed papers have the dual burden of being both hard to publish and having had to fail interestingly.

Improve this answer
5
  • 2
    It is not clear to me what the order of hardest to least hard to publish is in your list?
    – Stephanie
    Commented May 25, 2015 at 13:21
  • @Stephanie They're all "I can't be bothered to make this into a paper", rather than difficulty in submitting, and thus likely vary based on your motivation.
    – Fomite
    Commented May 25, 2015 at 19:53
  • This part "Well powered (big) studies that find what people expect Poorly powered (small) studies that find what people expect Poorly powered studies that find the opposite of what people expect or null findings Well powered studies that find the opposite of what people expect" not numbered bit
    – Stephanie
    Commented May 26, 2015 at 1:41
  • @Stephanie Ah, that makes more sense. They're in rough order of difficulty, from least to most difficult to publish, though that may be field specific and is based on a slide I once saw in a presentation.
    – Fomite
    Commented May 26, 2015 at 16:34
  • 1
    I like this answer the most s it actually answers the question. There isn't anything preventing you publishing a failure, but it costs time, effort and money; demands your attention, and must be spun into something with a point. I would also add one more: you do not wish to give away too much about your research, if it stills shows promise for some kind of success. Sad but necessary.
    – kaay
    Commented Dec 6, 2015 at 11:23
36

It is not completely true that failures are not published. Lack of signals, or lack of correlation are published. The point is that everything that pushes knowledge forward is worthy of publication. That said, there are other factors you have to keep into account

  1. some failures are methodological, that is, you are doing something wrong. That is not a scientific signal. it's something you have to solve.
  2. knowing what doesn't work gives you a competitive advantage against other research groups.
  3. negative signals almost never open new fields. If they do, it's because they steered attention to find a positive signal somewhere else. You don't open a new cancer drug development if a substance is found not to have an effect. You close one. For this reason, negative papers generally don't receive a lot of attention, and attention from peers is a lot in academia.
Improve this answer
7
  • 15
    @Nunoxic: that's the point: letting others waste their time so that while they are busy figuring out what you already learned, you can beat them publishing before them. Do you think academia is a bunch of hippies ?
    – Stefano Borini
    Commented Mar 15, 2012 at 5:25
  • 9
    The best example of a negative result that was hugely important that I can think of is the Michaelson-Morley experiment that completely blew the lid off the old aether theory. Most negative results are not nearly that attention-grabbing. :)
    – Shinrai
    Commented Mar 15, 2012 at 16:53
  • 2
    @StefanoBorini I think that is rather pessimistic. I regularly see medical literature published showing no advantage / effect for medications and that is important. Generally speaking, I think alot of dead ends aren't published because dead ends are often assumed; promising results (whether positive or negative) still take a long time to finalize, time that could perhaps be better spent pursuing something more promising.
    – george
    Commented Mar 15, 2012 at 17:03
  • 1
    @Nunoxic - No the point of publishing is that it spurs donations and interest in your research. If you put a bunch of papers out there about how you screwed up and wasted millions or even thousands in grant and private funds because you screwed up people are less likely to fund your next research project and may pull funding for what you have going.
    – Chad
    Commented Mar 15, 2012 at 18:13
  • 5
    @StefanoBorini: Yes, actually, we are a bunch of hippies. Some of us, anyway.
    – JeffE
    Commented Mar 16, 2012 at 23:47
13

One of the consequences of not reporting failures is publication bias. It is a widely described phenomenon that is usually dealt with using meta-analysis. That is if the research concerns some quantifiable results, say some (linear) regression, usually statistical significance is desirable. If such results are not met, researchers sometimes try to adjust their methodology, models, data or whatever else in order to have more "publishable" results.

The problem of either adjusting of models, or complete withdrawal of the paper (the so called file drawer problem) was (and to a certain extent still is) a problem in medicine, as @Artem Kaznatcheev notes in the comment and resulted, as he adds, in registering of trials before publication. His sources may differ, but a paper describing it is e.g. Krakovsky (2004).

More generally, look at Stanley (2005) or Stanley (2008) for more information on publication bias and meta-analysis.

Improve this answer
8

Sometimes some failures give rise to new theories themselves. For example, there are impossibility theorems in mechanism design (Arrow's or Gibbard-Satterthwaite's) which establish the limitations of implementable mechanisms.

In that sense, failures alone may not be useful in a publication. People are interested to know why things failed or rather what category of experiments/theories would fail. But proceeding in this direction is often fraught with risk: it is easier to state a problem and solve it rather than to derive conditions on when the problem cannot be solved. The former is interesting to a wider class of audience than the latter.

So the bottom line: try to formalise or theorise your failures and see if a result looms; else move on to try and solve a worthier problem.

Improve this answer
8

I think this comes down to the time taken. I personally could not be bothered spending 2 days writing up and formatting a paper for negative data. It needs to be really simple to do and I either need to get some credit for it, or need to be mandated to do it.

There is a growing requirement from funders and mandates forcing researchers to make all of their research outputs available are only a matter of time. In the mean time, altmetrics can act as carrots to encourage researchers to share their data, even the negative stuff.

Improve this answer
3

If you have read a lot of literature and then realized that you are in a dead end, you can at least publish a review on all the methods and techniques you've learned.

Improve this answer
1

There is the Journal of unsolved Questions JunQ. They collect ‘null’-result research and open problems.

Improve this answer

You must log in to answer this question.