1

I am very interested in understanding the aspects of current social science research that are generally accepted as being problematic (by researchers and other commentators).

I regard problematic aspects as those which at least some academics regard as as sub-optimal and present as problems which a discipline (or academia in general) should endeavor to address.

The problematic nature of the concepts (i.e., names for practices) that I have listed below can be inferred from the references that I have linked to them; which draw attention to issues that the authors feel are detrimental to research (either in general or within a specific area of inquiry).


Problematic practices:

Authoring practices

  1. Plagiarizing: researchers using another writer's words without proper citation (Loui 2002)

  2. Self-plagiarizing: researchers reusing their own work without proper citation (Loui 2002)

  3. P-hacking: researchers engaging in different processes until they find significant results (Simmons et al. 2013)

  4. Harking: researchers hypothesizing post-results so that they appeared to have predicted something you they did not (Kerr 1998)

  5. Data dredging: researchers searching through data to find anything that is significant (Smith and Ebrahim 2002)

  6. Underpowered studies: researchers conducting studies with research designs which lack sufficient power (Maxwell 2004)

  7. Lack of relevance: researchers conducting research that has little practical relevance (Bolton and Stolcis 2003)

  8. Lack of studies which replicate other studies: researchers conduct too few studies which attempt to replicate prior findings (Ioannidis 2005)

  9. Selective publishing of results: researchers running multiple studies and only reporting the most favourable ones (Turner et al. 2008)

  10. Unjustified self-citation: researchers citing their own work without valid reason (Gami et al. 2004)

Journal practices

  1. Publisher paywalling: journals make much current research [some publicly funded] becomes paywalled and inaccessible to most people (Teplitskiy et al. 2015)

  2. Dissemination delay: journals' review cycles are too slow and delay publication (Smith 2010)

  3. Editorial favoritism: journal editors may be biased toward accepting certain researchers research (e.g., due to familiarity, or prestige) (Yoon 2013)

  4. Significance favoritism: journals have excessive preference for significant results: Most journals will generally only accept research with significant results (Fanelli 2011)

  5. Self-citation favoritism: journals have excessive preference to publish papers that cite prior publications (Tighe et al. 2011)

  6. Blind faith in peer review: Despite being almost universally adopted by prestigious journals, peer review doesn’t result in higher quality research and has many known flaws (Smith 2010)


My question:

Which concepts (if any) from the list above would you modify, add, or remove?

For instance, do any of the concepts mentioned have alternate names that I should know about (e.g., data dredging and harking are similar), or better, more accepted names that I should use instead?

Similarly, there any concepts (i.e., practices) that I have failed to include in my list or any which you think should be removed from the list? Maybe some research says that actually a given practice is good for academia, or there is additional research which highlights issues that I do not mention?

If you, based on your personal experiences, or opinions, can rule out, or contribute even a single concept then that would be a very valuable answer.

It would be an even more valuable answer if you could provide evidence (e.g., a published source) to argue why the concept is, or is not, considered to be problematic within social science research.

Thank you :)


References:

Bolton, M. J. and Stolcis, G. B. (2003) 'Ties that do not bind: Musings on the specious relevance of academic research', Public Administration Review, 63(5), 626-630.

Fanelli, D. (2011) 'Negative results are disappearing from most disciplines and countries', Scientometrics, 90(3), 891-904.

Gami, A. S., Montori, V. M., Wilczynski, N. L. and Haynes, R. B. (2004) 'Author self-citation in the diabetes literature', Canadian Medical Association Journal, 170(13), 1925-1927.

Ioannidis, J. P. (2005) 'Why most published research findings are false', PLoS medicine, 2(8), e124. Kerr, N. L. (1998) 'HARKing: Hypothesizing after the results are known', Personality and Social Psychology Review, 2(3), 196-217.

Loui, M. C. (2002) 'Seven ways to plagiarize: Handling real allegations of research misconduct', Science and Engineering Ethics, 8(4), 529-539.

Maxwell, S. E. (2004) 'The persistence of underpowered studies in psychological research: causes, consequences, and remedies', Psychological Methods, 9(2), 147.

Simmons, J. P., Nelson, L. D. and Simonsohn, U. (2013) 'Life after p-hacking', in Meeting of the Society for Personality and Social Psychology, New Orleans, LA, 17-19.

Smith, G. D. and Ebrahim, S. (2002) 'Data dredging, bias, or confounding: they can all get you into the BMJ and the Friday papers', BMJ: British Medical Journal, 325(7378), 1437.

Smith, R. (2010) 'Classical peer review: an empty gun', Breast Cancer Res, 12(Suppl 4), S13.

Teplitskiy, M., Lu, G. and Duede, E. (2015) 'Amplifying the impact of Open Access: Wikipedia and the diffusion of science', arXiv preprint arXiv:1506.07608.

Tighe, P., Rice, K. J. and Gravenstein, N. (2011) 'Artifactual increase in journal self-citation', Anesthesia & Analgesia, 113(2), 378-382.

Turner, E. H., Matthews, A. M., Linardatos, E., Tell, R. A. and Rosenthal, R. (2008) 'Selective publication of antidepressant trials and its influence on apparent efficacy', New England Journal of Medicine, 358(3), 252-260.

Yoon, A. H. (2013) 'Editorial Bias in Legal Academia', Journal of Legal Analysis, 5(2), 309-338.

closed as too broad by corey979, Jon Custer, user3209815, gman, Scientist Dec 18 '18 at 17:59

Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer. Avoid asking multiple distinct questions at once. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.

  • 7
    While I haven't downvoted your question, because I think that it is a good and interesting question in general (and I applaud your comprehensive - for a question, that is - lists and references), I think that it is not a good fit for Academia.SE. Two reasons: 1) it is pretty broad; 2) adding, removing or modifying practices you've listed require specifying criteria for assessing those operations, which you haven't provided (without criteria, those decisions become very subjective - actually, they are most likely quite subjective even with criteria). – Aleksandr Blekh Jan 20 '16 at 4:26
  • 1
    Hi Aleksandr. Thank you for your comment. I can understand your concerns even if I do not fully agree with them. In my humble opinion, the question, while not perfect, is not overly broad. I am clearly looking for answers that can increase my relevant knowledge in this area of academic practice and result in the modification of my list. Additionally, while some concepts within the list may be unclear, I think that they are generally quite clear. Finally, even if it is not a perfect fit, I don't think there is a better venue to get answers for this question than Academia.SE. – Peter Slattery Jan 20 '16 at 4:52
  • 2
    Hi, Peter. You're welcome. I understand your points. Please take my words with a grain of salt - I might be right only partially or completely incorrect. Also, I quite agree with your last point, though you can also try Quora - some answers there are very good (you can even use Ask to Answer feature to select experts in a field). – Aleksandr Blekh Jan 20 '16 at 6:07
  • 2
    Also, you are almost certainly missing an item with regards to data forging, i.e., partially or completely making up your data. – xLeitix Jan 20 '16 at 6:35
  • 2
    "Fabrication" is more common than "forging", I think - you'll find plenty of discussion using that term. See, eg, ncbi.nlm.nih.gov/pmc/articles/PMC2685008 – Andrew Jan 22 '16 at 13:37
5

I would add:

Disguising political opinions and agenda as scientific facts.

The majority of scholarly work in social science has political motivations or implications. Too often the tools of experimental sciences (statistics mostly, but there are other examples) are used to legitimate what is really the author's political opinion.

Possibly, the source of this unwanted, if not fraudulent, behavior, is the push from funding sources to evaluate social sciences output the same way that experimental and STEM research is. Hence the absurd push to use statistics and show "significance" for issues were they are really not indicated.

I would remove:

Publisher paywalling

Although unfashionable, that is an efficient way of funding quality publishing.

Also, for non social-science fields I think what you refer to as "data dredging" is not problematic, and actually an interesting approach. There is growing interest in computer programs mining large data sets in search for clusters, co-variance and correlation. Although it would then be dishonest to obfuscate the fact that the relationships were found that way, of course.

  • Thank you for your answer Cape Code. Do you know of any paper which provides support for the claim that "Disguising political opinions and agenda as scientific facts" is regarded as problematic within social science? I appreciate your option in relation to the removal of "publisher paywalling" - I will take that view on board. Do you know of any paper which provides support for the claim that Publisher paywalling is needed to fund quality publishing, and thus suggests that it is generally regarded as a bad thing? – Peter Slattery Jan 21 '16 at 5:29
  • 1
    @PeterSlattery there's an enormous literature discussing the economics of scholarly publishing, of which Teplitskiy is only the tip of the iceberg. It's widely accepted that non-paywalled literature is a nice thing to have, but this is not quite the same as saying paywalled is inherently problematic – Andrew Jan 22 '16 at 13:28
  • 2
    @PeterSlattery incidentally, the Teplitskiy paper has some pretty major limitations - see notes at meta.wikimedia.org/wiki/Research:Newsletter/2015/… - so I'd try looking at a broader review than just that one example :-) – Andrew Jan 22 '16 at 13:30
2

In my opinion "harking" and "data dredging" are sufficiently dissimilar to merit separate points. You could run the tests you had planned to run (therefore probably not "data dredging"), find results that seemed quite opposite to what you hypothesized, and then change your hypothesis when you saw it didn't line up with your initial predictions (therefore "harking"). So they're both problematic practices, but different enough to warrant separation.

I do think however "data dredging" and "P value hacking" are essentially the same thing. Both involve going way, way off the path you'd originally planned for your analysis and scrambling around to find literally anything significant, even if you had not made a single prediction about any of the tested relationships. If I had to pick one name though I'd say "data dredging," because "P value hacking" makes me think more of outright falsifying p values... which is a different issue from running a million random tests until something produces a p value you like but didn't tamper with.

Not the answer you're looking for? Browse other questions tagged or ask your own question.