67

I have written a short paper (six pages) in which I empirically analyse publicly available data. Both the data and my analysis are genuine and without (intentional) errors. Still, by using statistical and argumentative tricks, the paper somewhat convincingly arrives at a conclusion that is obvious nonsense.

My goal is to demonstrate by vivid example how easily this can happen (intentionally or unintentionally) even when obeying the typical scientific procedure and rigor. That message is directed rather at the "interested public" than at the most academic readers. It is not a goal to fool anyone (so it's not a hoax paper), and I have included a "preface" that clearly states the circumstances.

Now my question is: Where should/can I publish such a paper? I thought about a parody journal like the Journal of Irreproducible Results at first, but it does not seem to exist anymore and also its scope was not 100% fitting. Should I try a "real" journal instead and hope for the editor's humor? Or should I just upload the paper to ResearchGate? My "criteria" are:

  • It would like it to be published in such a way that I can tell everyone it was accepted by "prestigious and well-known journal XY" (regardless of whether the journal is really prestigious), rather than just "a website".
  • The journal should be open-access or similar so that everyone can read the paper.
  • It should not be a predatory journal, because I do not want to harm my real academic reputation.
  • I do not want to pay publication fees, at least not high fees.
  • It should not take a year or so until the paper is published.

Thematically, the paper addresses a question that could be called a politics issue, so it might fit in a corresponding journal. I also thought about journals that deal with good/bad scientific practice in general.

Background: In the recent months to years, I had to read a lot of pseudo papers (mere PDF files I mean that luckily have not been published) about, well, what I and many other people would call conspiracy theories. The main content of such a paper usually was an empirical analysis that resulted in completely absurd conclusions.

My job was to find errors in these analyses and to refute them, but, as always, this proved to be very difficult. However, this is something that people outside academia often do not (or do not want to) understand... So I decided to turn the tables and write an even more obvious junk paper that nevertheless no one will be able to refute.


As clarification, I'm rather asking in which "category" to publish such papers in general, not concrete journals (except journals like Improbable.com, which however form a category on their own).

Some have expressed concerns that the paper despite all measures will be taken seriously by some groups. I can understand this very well, but my paper is actually about a theory that itself has been made up exactly for the purpose of demonstrating how easy it is to generate believable junk science.

5
  • 2
    look at this paper: jstor.org/stable/… The Spatial Distribution of the Montane Unicorn was published in a top ecology journal Apr 4 at 13:01
  • 1
    Have you also considered publishing the paper on a blog, or making it into a video presentation which links to the paper?
    – Taw
    Apr 4 at 13:02
  • 39
    Frame challenge: make a serious, formal paper about misusing and misinterpreting scientific data, and use your "parody" study as part/demonstration of it
    – Josh Part
    Apr 4 at 15:24
  • 1
    Various additional exemplars, answers, and discussions about whether this is a good idea have been moved to chat. Please see this FAQ before posting another comment, and remember that we can only move comments to chat once.
    – cag51
    Apr 6 at 9:16
  • Now I'm intrigued. Please share the paper here if it is ever available to the public. :)
    – fgysin
    Apr 19 at 9:44

10 Answers 10

62

The sort of paper you mention is very valuable to e.g. use in education to show that statistical analyses cannot be used without also understanding the underlying data. There are several papers like this published with the expressed purpose of making a point about over-interpretation or relying on statistical numbers alone to describe relationships.

The first that comes to mind is Anscombe (1973) (see wikipedia for a picture of the Anscombe quartet) who produced four data sets that are statistically identical (with reference to linear regression statistics) but when plotted indicate four very different types of relationships. The point of this paper was to show the power of graphical representation to complement statistical measures.

Another example is Collins (1965) which shows a correlation between sun spot activity and the stock exchange. Although the analysis is not described in detail regarding the physical relationships between the parameters it is an indication that spurious correlations may arise if one really looks for them.

A third example concerns the issue of p-values. Letters by Baker (2016) and Karpen (2017) can serve as an example voicing the issue of over-interpreting the value of p-values in studies.

As for journals to convey messages like this there are two options as I see it. One is to go for an obviously extraneous journal such as Improbable Research (https://improbable.com/). In the past there was also the Journal of Irreprodicible Research. A second option is to contact the editor of a statistics journal to enquire whether or not a paper such as the suggested could be published in some format in such a journal. To submit the paper as a regular paper is probably not a wise idea but trying to locate an avenue supported by journal editors would be a useful addition to our understanding of limitation of statistical methods. In addition there are journals specifically targetting higher education where publishing around problems as these suggested may be welcome for educational purposes.

References

Anscombe, F.J. 1973. Graphs in Statistical Analysis. American Statistician. 27 (1) 17-21. doi: 10.1080/00031305.1973.10478966

Baker, M. 2016. Statisticians issue warning over misuse of P values. Nature. 531, 151

Collins, C.J. 1965. An Inquiry into the Effect of Sunspot Activity on the Stock Market. Financial Analysts Journal 21 (3) 45-56.

Karpen, S.C. 2017. P-value problems. American Journal of Pharmaceutical Education. 81 (9) 93.

4
  • 4
    Also: Theory of the stork
    – Roland
    Apr 4 at 11:09
  • 4
    Here's another example to consider including in your nice answer: jstor.org/stable/… Apr 4 at 13:18
  • 12
    If you like Anscombe's quartet, you'll love the Datasaurus Dozen. Apr 4 at 15:44
  • 15
    "four data sets that are statistically identical"... I would probably reword this as "four data sets with identical summary statistics". At least to me, "statistically identical" implies you cannot come up with statistics that would distinguish them, which doesn't seem true. (Not that my alternative is airtight either, but I feel like it's closer.)
    – user541686
    Apr 5 at 4:41
56

The BMJ is a very serious medical journal. It has published two studies related to the use of parachutes when jumping off a plane.

The first one is a review paper. It attempted to find RCT (randomized controlled trials) of parachute use, found none, and "concluded" that there is no evidence for the use of parachutes.

The second one actually did a trial. The abstract confidently states that intervention (parachute) vs. control (no parachute) did not change survival at all. The reason for that is that survival rate was 100% in both groups, because the jumps were performed from a small landed aircraft ("mean altitude 0.6m, mean speed 0km/h").

Those papers actually have serious messages. The first one is about equipoise: there has been no gold-standard trials of parachutes, because nobody seriously doubt they work; it criticizes the rigidity of a "RCT or nothing" mentality. The second is about undue emphasis on statistical methods and abstract-skimming, when important methodological details are buried in the paper. (Also, to me at least, they’re hilarious to read.)

There is thus precedent to publish parody papers in serious journals, fairly recently (2008 and 2018).

4
  • 9
    Ah, but what were the standard deviations of the speed and altitude? Eh? Bet'cha never thought of that! :-P
    – einpoklum
    Apr 4 at 9:24
  • 3
    @einpoklum - for a sufficiently small airplane, one might need Heisenberg as a co-author...
    – Jon Custer
    Apr 5 at 20:24
  • @einpoklum Zero. All jumps were performed under the same conditions, changing only whether they wore a parachute or not. However, it wasn't blinded. Apr 5 at 23:43
  • @LorenPechtel ah but what's the accuracy of that zero?
    – user253751
    Apr 6 at 12:36
29

If sufficiently useful, papers like this are published in the same journals where their message would be influential. An example would be:

Bennett, C. M., Miller, M. B., & Wolford, G. L. (2009). Neural correlates of interspecies perspective taking in the post-mortem Atlantic Salmon: an argument for multiple comparisons correction. Neuroimage, 47(Suppl 1), S125.

This is a critique of neuroimaging methods published about fMRI of a dead salmon. It's published in a popular journal for fMRI-based and similar research.

7

I know of one such example, an investigation of correlations between star signs and health conditions. It got somewhat famous after the German "newspaper" BILD fell for it, despite being very much not written as a hoax paper (so be wary, no nonsense is so obviously nonsense that no idiot will fall for it).

The paper was published in what I assume to be a regular epidemiology journal: Testing multiple statistical hypotheses resulted in spurious associations: a study of astrological signs and health

8
  • 2
    Interesting, but not an answer, actually.
    – Buffy
    Apr 3 at 13:01
  • 7
    @Buffy It shows that there is precedence for "publish in a regular journal". If we read the question as "point me to a particular journal I could publish in", we'd have to close it as a shopping question.
    – Arno
    Apr 3 at 13:09
  • 1
    I'm not advocating for a scientific basis for star signs, but I think it seems reasonable that star signs could correlate with some health conditions? To my understanding they're based on your birth month and I think birth month could be a predictor for some health conditions; eg. maybe something like seasonal affective disorder, or also I've read that if you're young for your school year group it can have a negative impact on learning social skills which could impact health later in life
    – Joe
    Apr 3 at 20:55
  • 1
    @Buffy If the question should be understood as "recommend a journal for me to publish this paper in" then I agree, it'd seem like a shopping question, but by default I'd understand it as "what kind of journal would make sense to publish this paper in, and/or how can I find such a journal", and in that case I don't think it's a shopping question. (Although of course it would be better if the question explicitly said that)
    – David Z
    Apr 3 at 22:11
  • 2
    Similar: xkcd.com/882.
    – hlovdal
    Apr 6 at 13:46
7

This is a pointless and potentially damaging exercise.

YouTube and the web are filled to the gills with serious analyses debunking accidentally or deliberately badly analyzed data or claims. And you know what? They're basically preaching to the choir because the people who want to believe the claims still believe them and the people who don't agree with the claims are not being told anything they don't know already.

The people you'd reach in a even vaguely professional journal already know how to do a proper analysis. Despite that, some of them will, for whatever reasons of personal bias, happily make conclusions they can easily see are false by e.g. ignoring inconvienent data or taking the result out of context or whatever.

I'm afraid you won't achieve anything by publishing an example of such a paper because unfortunately we have an almost endless supply of such material already out there and being greedily consumed by a target audience that doesn't want to believe it's wrong and won't listen to any proof or educational material explaining why it's false.

Remember you live in a world where the media "balance" debate by putting on the tin-foil hat brigade and giving them equal (if not more) time as actual mainstream scientists. People select their news sources as ones that confirm what they already believe. Who is going to actually benefit from your paper in such a climate?

In addition to this, the reality is publishing such a paper would probably do you more harm than good. It would be forever attached to your name and the "obviously" false conclusion would be leapt on by people who like it for their own reasons. Indeed the paper could later be cited by people who want to debunk one of your own legitimate papers by showing an example of how you messed up before and cannot be trusted.

So it's pointless and potentially personally damaging.

I would not recommend doing this.

18
  • 4
    Your conclusion is correct, but I do not see the relevance of most of it. The question is not about YouTube or popular media. Apr 3 at 22:41
  • 31
    The people you'd reach in a even vaguely professional journal already know how to do a proper analysis → well, you probably did not see statistical analysis in some areas of biology or economics. A friend of mine asked me to review her PhD thesis for statistics (she was a hard-core biologist) and I am traumatized to this day. She is a well know professor now.
    – WoJ
    Apr 4 at 10:00
  • 3
    "They're basically preaching to the choir because the people who want to believe the claims still believe them and the people who don't agree with the claims are not being told anything they don't know already." I completely disagree. I know people who have been convinced by conspiracy-theorists, and who then have been successfully "unconvinced". This would not happen if debunkers self-censored by fear of "only preaching to the choir".
    – Stef
    Apr 4 at 17:30
  • 7
    "The people you'd reach in a even vaguely professional journal already know how to do a proper analysis." Not every researcher is a statistician. I work in a research lab and I can tell you that even seasoned researchers can be relatively easily fooled by a well-done false analysis. Also, we have a "popularisation of science" journal that regularly gets our analyses wrong, because of miscommunication between the journalist and the researcher. I don't blame the journalist - often it is the researcher who wasn't wary enough of how their results could easily be misinterpreted.
    – Stef
    Apr 4 at 17:35
  • 8
    @StephenG-HelpUkraine "Professionals really do know how to do a proper analysis, the fact is some do not for reasons ranging from personal bias to carelessness" - That is very counter to my experience in science. Many professional academic scientists simply do not have sufficient background in statistics, and it's not because they are biased or careless, but because A) They aren't trained in it, and have so much else to learn that they haven't made time, and because B) For a surprising amount of science, quantitative statistics don't matter.
    – Bryan Krause
    Apr 4 at 21:36
6

If you work at a university, would a university periodical, departmental bulletin or similar be an option? At least here in Japan, these are plentiful (The Kobe City University Journal, Shinshu Studies in Humanities, The Keio Journal of Arts and Letters, etc.) and while lesser to other journals are somewhat respected. Quick to come out, no fees, open access (university repository), you can tell people it's a 'respected journal'... it seems to fit your requirements.

1
  • 1
    Thinking outside the box. Most of this stuff is shared via web link anyway. When is the last time you cared which server your paper comes from, aside from the paywall and possible concerns of legitimacy? Apr 6 at 17:56
3

...by using statistical and argumentative tricks, the paper somewhat convincingly arrives at a conclusion that is obvious nonsense.
My goal is to demonstrate by vivid example how easily this can happen (intentionally or unintentionally) even when obeying the typical scientific procedure and rigor.

If your intent is serious and you mean to produce useful science then I'd suggest abandoning the specific example you've proposed and instead focus on generalizing the fallacy(ies) you relied upon to produce said nonsensical result.

You will probably discover that, in generalizing, you can produce a far more useful scientific result than simply examining a single specific case. Leaving it as an exercise for the reader doesn't really provide people with a tool, metric, or algorithm with which to gauge their own methods in search of similar fallacious methods or reasoning, and so has limited value.

Science is about first being intrigued by a counter-intuitive result, but finally is about breaking down that counter-intuitive result until you can come to a place of understanding. In so doing, you may also end up discovering an entire field of research that already focuses on the topic you're just now scratching the surface of. Your work may not be as novel as you think, and you may, in addition, be an entire lit-review away from even having such an idea for the paper.

3
  • 2
    Why not do both? Generalizations and examples are best when used in unison. The former highlights the patterns and the latter allows for something more tangible to root one's understanding in.
    – user110391
    Apr 5 at 0:49
  • 2
    @user110391 Fair enough, but the example should be the icing and not the cake.
    – J...
    Apr 5 at 0:51
  • "(...) but the example should be the icing and not the cake." Personally, I too believe that to generally be the case, but sometimes generalizations can get so abstract that a well-explained example is essential (thus making them the halves of the cake). This is probably not such a case though.
    – user110391
    Apr 5 at 1:12
1

Depending on your field, I think there's a good chance you may be able to submit such a paper to many (open-access, per your requirement) journals: just not as a research article. As an example, PLOS Biology offers an extensive list of article types published in the journal; looking through the options, your article might fit as a Perspective or an Essay. In mathematics, I recently read a work about citation practices published as a Viewpoint in the Mathematical Intelligencer; it didn't invoke data but was written in a satirical style that strongly reminds me of your goal. The "Applications and Case Studies" article type in the Journal of the American Statistical Association might even fit the bill (although I didn't read its open access policies). As a final case study, the Frontiers In family seems to have a large selection of article types from which you can use; I linked Frontiers in Applied Mathematics and Statistics above.

1
  • Thank you for citing my paper! Hope you enjoyed it! Jul 26 at 8:32
0

There is only one place to do this - the Journal of Irreproducible Results. It's a Science Humour and Parody magazine, so there is no danger that your parody will be taken seriously.

1
  • 2
    I would very much like to do so, but unfortunately the journal does not exist anymore it seems. The - in some unofficial sense - successor to it, the Annals of Improbable Research, rejected my submission because it did not fit the scope exactly.
    – Remirror
    Apr 4 at 7:08
0

Have you tought about a Philosophy of Science Journal? That might be an appropriate place to try.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .