24

As in the title, someone else's PhD student is doing things that are clearly wrong in terms of statistical methodology. Because it is a field of social sciences where statistical methodology is not so common and well developed, they are getting away with that, surprisingly.

At departmental seminars and colloquiums with more quantitative-oriented researchers (myself included), they receive criticism but the student in question seems to be having difficulties addressing it. The main advisor, who seems to not be very knowledgeable about quantitative methods (at least not beyond point and clicking his way through SPSS and getting tables/plots) apparently supports all that is done and sees no problem. Advice on how to improve the methodology and to seek the help of statisticians goes unaddressed. It seems that their modus operandi is to continue doing what they are doing and strategically choose journals where reviewers will not pick on their flawed quantitative methods.

I see two problems there. First, the student is receiving wrong advice and learning problematic habits/strategies for research. From an educational point of view, it is wrong to educate someone to do sub-standard research when it is possible to teach them the correct way of doing it. Second, it is problematic to submit and publish flawed research, even if it is just as part of the education of a PhD student. It is not the case that they understand the criticism but have valid counterarguments. It is rather that they either do not understand Statistics well enough to realise the problem, or that they do but choose to deliberately ignore the problem because it doesn't influence the prospects of publication in their field. In either way, the result is that substandard research is produced and published when it would be perfectly avoidable.

One of the co-advisors of the student agrees with me and our other colleagues regarding the methodology and the need for a methodological advisor (a statistician, perhaps), but both the student and the main advisor don't. What can be done in this situation? Should I just leave it be and treat it as one of the many other oddities in academia? Should I approach the main advisor or the student and try to convince them somehow? Is there a good way to improve this situation? Preferably without disturbing the harmony of our department (which is very harmonious, discussion friendly, and conflict-free). It is also worth adding that I'm junior faculty and would like to avoid career-suicide.

  • 14
    "It is also worth adding that I'm junior faculty and would like to avoid career-suicide." This seems to be important information :) my personal stance would be that somebody else's student would not be a hill I would wish to die on, but others may be more principled ... Anyway, I am definitely looking forward to the answers to this. – xLeitix Feb 16 '16 at 16:52
  • 4
    Is the co-advisor who agrees with you also junior faculty? If not, perhaps they can be the one to raise the issue. Also, as a co-advisor, it would probably be less contentious for them to give unsolicited advice. – bmurph Feb 16 '16 at 17:33
  • Perhaps that is that advisors battle, not mine then. In the spirit of avoiding conflict, is it a good idea to approach that advisor and get them to raise the issue? Will that make things between advisors sour? Perhaps that is somebody else's (their) problem after that? – Kenji Feb 16 '16 at 17:55
  • 2
    I was a CS postdoc on a physical educations lab for a while. I saw that exact problem, but with CS, not math/stat. I tried to help, pointing errors and possible solutions. I think it might have been a factor to getting canned a couple months later... Not the only factor, it was way more complex than that, but it didn't help. Some people believe so much on themselves that even the thought of being wrong, in other field, is unbearable... Hope it is not your case... – Fábio Dias Feb 16 '16 at 18:44
  • Well, I think I'm fine with being wrong :) I do hope that it is not their case, as their research is interesting and could definitely be published in more visible journals if they would only polish up the methods. Some of the best faculty in our department are doing math/stats and had great constructive advice (much better than mine), but were also ignored. – Kenji Feb 16 '16 at 18:52
14

I think that an important criteria in deciding what action to take is how severe the effect on the research quality actually is. Practices that look horrifying to an expert in the practice may actually have only a minor effect on the actual validity of scientific experiments that do not depend closely on the details.

I would recommend selecting a response based on how severe the likely issues caused by the sloppiness of statistical analysis actual is:

  • If the analyses are actually actively fraudulent (e.g., taken with intent to deceive), then you've got a responsibility to blow the whistle (though you must take care in confirming and in how you go about it in order to avoid destroying your career).
  • If the analyses appear likely to cause a major qualitative change in the interpretation, such that retractions would be appropriate, then it is reasonable to warn both your student and your colleague of the issue and to point out exactly how and why such issues are likely to arise. If they understand the issues and still choose to go in that direction, it is not your responsibility to save them from embarrassment and possible retraction.
  • If the analyses are incorrect and probably distorting results but the effects they are working with are strong enough that the distortions are unlikely to radically change the results, then you can point it out, but don't be surprised if they ignore you for the sake of expediency and "good enough" approximations---and indeed, they may be correct to do so!
  • Well, actually their procedures are just done with the wrong type of data, in a way that makes it impossible to even conclude anything. It is like they are 'not even wrong.' If I give too many details I may lose my anonymity, but it is something like estimating a duration model using historical time instead of time since the onset of risk (if the machine breaks in 2010 it worked for 2010 years) type of mistake.They do not estimate a duration model, but this is close to the type of mistake they make. – Kenji Feb 16 '16 at 18:11
  • @Kenji At the risk of being controversial: how much does their "not even wrong" actually matter? Please understand that I am not advocating for doing things the wrong way, but sometimes "proxy measures" and coarse approximations actually do well due to strong correlations with the proper measure. – jakebeal Feb 16 '16 at 19:27
  • 1
    I see the point of using coarse approximations and their validity, but in this case they are mostly drawing conclusions out of noise data. – Kenji Feb 16 '16 at 20:04
  • 1
    @Kenji Then it sounds like it would follow my middle case, and I would advise you to act accordingly. – jakebeal Feb 16 '16 at 20:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.