3

I published several medical meta-analysis papers in high-profile journals using advanced methodology that, to my eye, should be most interesting to many people in my field. However, I noticed that:

  • while statistical reviewers are very encouraging and interested, editors and reviewers would prefer more mainstream methodology and their comments often show a dire lack of understanding of the results
  • editors and reviewers aggressivity is proportional to the complexity of the analysis
  • ... and most importantly, those papers are very little cited compared to super mainstream works that I coauthor, even though far more effort is spent into all aspects of the paper, including vulgarisation of the results.

Ultimately, applied statistics should be a vessel for communication. So, should I revert to stone-age type mainstream methodology, even though results are in fact far weaker and less interpretable to the trained eye? Or should I persist with the hope that my methodology will be more appreciated in the future?

14
  • 2
    Why is this a interpersonal-issues? Commented Dec 31, 2023 at 23:51
  • 1
    @bryankrause I would like to argue that the career implications of my question are not a matter of personal values, and more advanced researchers would be able to answer from experience.
    – Raoul
    Commented Jan 1 at 0:19
  • 3
    @Raoul I am not sure you are aware of it but your comment/question oozes with condescension. As a researcher it is your job to effectively explain your work to others. I doubt the problem is that editors, reviewers and readers are too simple to understand your high level methods. Maybe you can provide an example of how your methods are so much more advanced than all of the other researchers in the "mainstream". From my experience, I do not believe you alone are doing something so different than all of the other researchers in the world. Commented Jan 1 at 0:44
  • 1
    @jasonwhite I'm not trying to say that I'm better than most. I'm asking whether I should keep using methods that are objectively better, but less understandable without a statistical background. Concrete example: network meta-analysis vs. Multiple paired traditional meta-analyses.
    – Raoul
    Commented Jan 1 at 0:52
  • 4
    @JasonWhite I'd disagree with you a bit on that...reviewers for medical journals are not necessarily that statistically sophisticated. Their training is in medicine, not statistics. Especially if they got their degrees longer ago, they've likely had at most a rudimentary statistics education, maybe just a single course. Everything else they have to learn on the job, and unlike other fields clinician scientists are often part time researchers, full time clinicians.
    – Bryan Krause
    Commented Jan 1 at 2:02

3 Answers 3

4

As a bit of background, I am a statistician and have visibility into psychology and medical research, e.g., theses my psychologist wife (with an affiliation in the local medical school) supervises, and papers her students and postdocs prepare. I absolutely endorse your observations that:

  1. Many bad statistical practices have become endemic in the medical and psychological fields. We regularly see examples at CrossValidated, from spurious dichotomization to questions about normal distributions for correlation analysis, both of which show a complete lack of fundamental statistics knowledge, but which are extremely common in published research, and yes, even in top journals.
  2. Unfortunately, most reviewers in these fields are subject matter experts, not statistical experts. They have neither the training, nor the time, nor the inclination to understand the statistical issues at hand.

Now, as others have answered, it appears that your job as a researcher is simple: just explain why you are using correct methods rather than wrong ones. Easy, right?

Unfortunately, that only works in theory. The established methods are established. If you analyze a continuous covariate, it is easy to explain that you did a median split to create two groups, or that you did stepwise model selection, and most subject matter experts will nod their heads and read on. That both practices are inappropriate is harder to see, and requires an argument. You do not want to be making that argument in a paper that is not methodological, just as you don't want to be making an argument for why your clinical trial was double-blind. This is not your focus!

So you don't make the argument, but rather go straight away to the more complex but correct analysis... and then the subject matter expert reviewer does not understand what you are doing and why, and recommends rejection, because you "are not following established methods" - without understanding that these methods are flawed in the first place.

I do not see an easy way forward here, and I have been frustrated with the way things are for 15-20 years now. The only possibility I see is to have that methodological paper that explains why the commonly accepted method is bad practice, and to name-drop that paper, possibly in a footnote - because, as above, you do not want to be justifying your choice of methods in your substantive (non-methodological) paper.

We analyzed our data using method foo.\footnote{Note that the commonly accepted alternative analysis using method bar is flawed, see Baz (2024).}

Sometimes, this may mean that you need to write that methodological paper yourself. Which may actually be a good thing, because now you have one methodological paper which uses your data as an illustration of your method, and a substantive paper that uses the more appropriate toolset.

1
  • 1
    Thank you for your input. That's pretty much what I had in mind, and my first supportive methodological paper is actually already started. It's interesting to get opinions from more advanced/specialized researchers though, and for that reason I'm unsure whether my question should be closed as opinion based, but c'est la vie! As an aside, I was pretty surprised at the irritation I generated in the process. Cheers!
    – Raoul
    Commented Jan 1 at 17:06
1

If your results can be obtained with simple tools, then do that. Otherwise..

(a) If simple tools don't suffice, use what you have.

however,

(b) If the more sophisticated tools provide insight into other problems in the field (or related fields) then it might be worth the effort to explain that along with the use of the tools. That insight might be more valuable than the results you obtain in any given study.

In math, for example, the proof of a theorem might be complex, but it might also be more important than the statement of the theorem proved provided that it gives insight into a class of problems that people consider important.

So, it is a judgement call. But don't make it complex for the sake of complexity or to show that you can handle complexity.

2
  • 1
    The remaining problem with b is that the insights provided have to be understood and accepted as such. Most clinical experts clearly don't see the value of what I'm doing, while statistical experts understand the benefits. Ultimately, the opinion of clinical experts is the only one that counts because they're the ones shaping the future of the field.
    – Raoul
    Commented Jan 1 at 0:25
  • Per my comment here, it is unfortunately usually not possible to explain the issues with a particular bad practice in the context of a non-methodological paper. Commented Jan 1 at 15:51
-1

Originally I thought it would suffice to only reply via comments but the more I thought about it I needed to make a longer answer, for others that may see this post. @Raoul your original post and your responses to comment replies not only show a lack of respect for your peers but also an abandonment of a core responsibility as a researcher to be able to effectively communicate your science. I have met plenty of researchers that are pushing the envelope in their respective areas but I have NEVER heard one of them complain about editors, reviewers and/or readers being unable to understand their brilliance. It is our job to communicate our work to others. We do that in our courses. We do that in our labs. We do that with collaborators. We do that to dept heads and other leaders. We do that to funding agencies. We do that at conferences and symposia. We do that in manuscripts. WE do it with or family and friends. If you are finding that to be a problem, it is not the audience. It is you. It is a lack of proper training or adequate effort and suggests you should focus your energy on refining your scientific communication skills. You can start by reaching out to collaborators that are clinicians to get their input but I would suggest you approach the conversation from a position of humility and respect for others in the scientific community. It is okay to be confident and excited about your work but don't let your ego get in the way of appreciating the work of others.

8
  • 1
    I don't really understand why you're so riled up by the idea that (clinical) editors and reviewers aren't statistical experts. It's true for most journals, and doesn't diminish the value of their clinical opinion that is, ultimately, the only one that counts. People don't have time to be experts in everything. Additionally, where did I state my methods are so exceptionally advanced? They are more or less what statisticians would deem 'appropriate', and well published. It's just that in medicine, mainstream really is low quality, statistically speaking.
    – Raoul
    Commented Jan 1 at 7:54
  • 1
    Rereading your comms, my impression is you're from a field where everyone knows what they're doing, hence your impression. Medicine is very different, as most papers are authored with approximately zero statistical knowledge by professional clinicians who write papers on their free time. So readers, reviewers and editors often are not statistically proficient. That's why we're having such a replication crisis, have special statistical reviewers, and why it's questionable whether doing more advanced analysis is appropriate for communicating with the readers at large.
    – Raoul
    Commented Jan 1 at 8:58
  • 2
    -1. Raoul is not discussing "readers being unable to understand their brilliance". As a statistics person somewhat active in a medicine-adjacent field, I absolutely endorse their impression that many reviewers and editors in these fields continue to use objectively wrong statistical methods, and will happily reject papers that use more appropriate statistics because they do not understand these more appropriate methods. Examples abound at CrossValidated. ... Commented Jan 1 at 15:48
  • 2
    ... It is unfortunately not easy to explain why one uses an advanced method and why an established method is wrong, especially not in the context of a paper that is not methodological per se. So your recommendation "to effectively communicate your science" is correct in theory, but misses the mark in the context of unfortunately entrenched bad statistical practice plus reviewers and editors that are subject matter experts and not statistics experts. Commented Jan 1 at 15:49
  • 1
    @jasonwhite I'd argue that "... something a newly minted PhD should be able to figure out" isn't very compatible with the position of humility you wanted me to adopt. I don't think it was useless to ask for the advice of more experienced people. I think I understand where you're coming from, but honestly I think you jumped up the gun and we're a bit judgmental. Medical science is NOTHING like hard science research.
    – Raoul
    Commented Jan 1 at 17:22

Not the answer you're looking for? Browse other questions tagged .