We noticed a problematic but influential preprint, wrote a commentary paper to criticize and rebut it, but were not able to post the comment to any public preprint repository. In this situation, what can I do against bad science?
Note: Some focus on our wording being too harsh, even accuse us of "personal attacks". In fact, we enlisted the help of many proofreaders to ensure that we did nothing like these, especially when the situation was so politically sensitive. Most feedback says that we were too friendly.
On 8 Jun 2020, a preprint is posted on Havard institutional repository "DASH". It suggests, based on satellite imagery of hospital parking lots and internet search trends, that COVID-19 outbreak started in Wuhan, China in August 2019. The officially documented first case was on 1 Dec 2019.
This study attracted significant media coverage and political attention (cited by Trump). But from a scientific point of view, its method and argument is really ridiculous. It is criticized by many, but mostly on social media.
We decide to write a commentary paper to criticize the study. To explain our concern and motivation, here is a summary of the conversation between me (A) and a colleague (B) when they tell me about this study:
A: So did they submit the paper to a journal? It will be rejected.
B: They don't need to if the purpose is media attention.
A: But we do have things like PubPeer that allow people to criticize anyway.
A: OK ... DASH does not assign DOI, so the paper is not indexed by PubPeer ...
B: Yeah, that's how they announce shitty studies now. Post online, news release, then who cares about publication.
A: Not necessarily. We can always write a manuscript criticizing it, and post it to arXiv.
B: Can we? Let's do it.
So I organized a team and drafted a paper. We fact-checked many information, rerun the statistical analysis to spot the main fallacy, and asked many people (academics and journalists) for proofreading. We identified many problems, including statistical fallacies and cherry-pickings.
Our original plan: posting our commentary to arXiv so that everyone sees it. Then we wait for the original paper. If they get published in a journal (to my surprise), we will submit our comment to the same journal.
As a mathematician by training, I'm a big fan of preprints, and a heavy user of arXiv. I usually post to maths category, occasionally to condensed matter physics category. I know that arXiv accepts commentary papers like ours. Since we are criticizing a paper about "digital epidemiology", I checked related arXiv preprints, and determined that "stat.AP" is the most suitable category.
But to my surprise, our submission is put on hold by arXiv. Then the major category is changed to "None". So I wrote to arXiv moderators to explain our choice of category. Then I get the following "rejection letter":
Our moderators have determined that your work would benefit from additional review and revision that is outside of the services we provide. Our volunteer moderators are not referees and do not provide reviews or other detailed feedback for improvement of submissions with their decisions.
As a result, we have removed your submission. Please instead send your paper to a conventional journal for the necessary reviews.
Please do not resubmit this paper without seeking permission and obtaining a positive response. Resubmission of removed papers may result in loss of submission privileges. We will reconsider this decision if your work is published or accepted with a resolving DOI (Digital Object Identifier) or link to the journal's website showing the status of the work.
This was very shocking to me, but I will refrain from commenting on this decision.
We then think about medRxiv and bioRxiv, but they made clear in their policies that they do not accept commentary papers.
We ended up posting the commentary on my personal academic website.
Announcing scientific studies by press release is becoming a trend now.
Imagine someone, out of malicious motivation, decided to post misinformation in the disguise of an "academic paper". To avoid peer review, they do not plan to submit the "paper" to any journal. To avoid PubPeer, they intentionally avoided arXiv, medRxiv, or anything that assigns a DOI. Then they promote their "paper" in media, and gained significant attention. They might receive critics, but only on social media. The critics are mixed up with other misinformations and conspiracies, and will not be taken serious.
In this situation, what can I do as a concerned peer? If I made effort to write a detailed critical analysis, how is that helping if no preprint repositories accept it?