Someone else's PhD student is not getting proper advising on methods. Should I interfere? How?

Question

As in the title, someone else's PhD student is doing things that are clearly wrong in terms of statistical methodology. Because it is a field of social sciences where statistical methodology is not so common and well developed, they are getting away with that, surprisingly.

At departmental seminars and colloquiums with more quantitative-oriented researchers (myself included), they receive criticism but the student in question seems to be having difficulties addressing it. The main advisor, who seems to not be very knowledgeable about quantitative methods (at least not beyond point and clicking his way through SPSS and getting tables/plots) apparently supports all that is done and sees no problem. Advice on how to improve the methodology and to seek the help of statisticians goes unaddressed. It seems that their modus operandi is to continue doing what they are doing and strategically choose journals where reviewers will not pick on their flawed quantitative methods.

I see two problems there. First, the student is receiving wrong advice and learning problematic habits/strategies for research. From an educational point of view, it is wrong to educate someone to do sub-standard research when it is possible to teach them the correct way of doing it. Second, it is problematic to submit and publish flawed research, even if it is just as part of the education of a PhD student. It is not the case that they understand the criticism but have valid counterarguments. It is rather that they either do not understand Statistics well enough to realise the problem, or that they do but choose to deliberately ignore the problem because it doesn't influence the prospects of publication in their field. In either way, the result is that substandard research is produced and published when it would be perfectly avoidable.

One of the co-advisors of the student agrees with me and our other colleagues regarding the methodology and the need for a methodological advisor (a statistician, perhaps), but both the student and the main advisor don't. What can be done in this situation? Should I just leave it be and treat it as one of the many other oddities in academia? Should I approach the main advisor or the student and try to convince them somehow? Is there a good way to improve this situation? Preferably without disturbing the harmony of our department (which is very harmonious, discussion friendly, and conflict-free). It is also worth adding that I'm junior faculty and would like to avoid career-suicide.

Answer 1

I think that an important criteria in deciding what action to take is how severe the effect on the research quality actually is. Practices that look horrifying to an expert in the practice may actually have only a minor effect on the actual validity of scientific experiments that do not depend closely on the details.

I would recommend selecting a response based on how severe the likely issues caused by the sloppiness of statistical analysis actual is:

• If the analyses are actually actively fraudulent (e.g., taken with intent to deceive), then you've got a responsibility to blow the whistle (though you must take care in confirming and in how you go about it in order to avoid destroying your career).
• If the analyses appear likely to cause a major qualitative change in the interpretation, such that retractions would be appropriate, then it is reasonable to warn both your student and your colleague of the issue and to point out exactly how and why such issues are likely to arise. If they understand the issues and still choose to go in that direction, it is not your responsibility to save them from embarrassment and possible retraction.
• If the analyses are incorrect and probably distorting results but the effects they are working with are strong enough that the distortions are unlikely to radically change the results, then you can point it out, but don't be surprised if they ignore you for the sake of expediency and "good enough" approximations---and indeed, they may be correct to do so!