5

I heard on Twitter today about a paper stating that after the Snowden leaks, people looked up terrorism-related topics less on Wikipedia.

I'm not worried about deliberate fabrication of data, but I'm worried that poor quality statistical analysis may have happened. I'm not expert enough to look at the paper's statistical analyses myself and determine whether they're appropriate, so instead I want to know whether rigorous peer review has taken place.

The journal involved, the Berkeley Technology Law Journal, doesn't seem to mention peer review either on its about page or the page about the journal. The page about submissions says "Evaluation involves scoring by multiple different editors, and rarely lasts fewer than 3-4 weeks.", which suggests that it's evaluated only by editors, not by academic peers. Also, evaluation and scoring sounds different to what I'd expect to happen in peer review - it sounds like they're only evaluating whether it's a good fit for the journal and of enough interest.

How do I determine if a paper has undergone anonymous peer review of its statistical analysis, short of contacting the editor of the journal?

Related question: How do I recognise a peer-reviewed journal article? but the answers seem to focus on science related subjects, as opposed to the humanities.

  • 2
    "BTLJ’s eponymous journal is a student-produced publication covering emerging issues of law in the areas of intellectual property, high-tech and biotech" (emphasis mine). If it is a student-produced publication, it is probably not your most trustworthy outlet of science :) – mmh Apr 30 '16 at 15:06
  • 4
    @mmh That is not true for the field of law, in which many important journals are curated by law students. – jakebeal Apr 30 '16 at 16:22
  • 3
    @jakebeal That's interesting! I did not know that. Thank you for pointing this out. Why is it so? – mmh Apr 30 '16 at 17:04
  • 2
    @mmh I don't know why it developed so differently in law than in other disciplines, but it has historically been the case in the United States: see this Wikipedia article, for example. If you search US law journal rankings, you'll find they're almost all student-run journals. – jakebeal Apr 30 '16 at 18:28
8

Let's tease apart two issues here:

  1. Has a paper undergone peer review of its technical content (which will include the statistical content)?
  2. Should you believe the statistics in the paper?

If it's in a well-respected peer-reviewed journal, then you can assume that it's undergone reasonable peer review. That means it's unlikely that the content is blatantly and obviously crazy, and that in fact it's likely to be fairly solid. It does not mean, however, that it is correct, particularly when it comes to something as subtle and frequently mis-applied as statistical analysis.

Bad statistical analysis is extremely common in every field of science, for a number of different reasons. Ignorance is a really big one, though, and one that's likely to be shared by the peer reviewers. Thus, if you want to know whether to trust statistical analysis, you need to look at it yourself.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.