When going through research papers statistical studies of political science phenomena, how do I know which sources I can trust?

Question

So far, I've learned mostly from textbooks but I'd like to read actual research papers and gather information from the sources themselves. My question is: how can I assess, as a stranger to the field, what research material can be considered relevant to a certain topic I have in mind and which is not?

The reason I ask specifically for political science is that if I were to talk about something like Mathematics, then I could probably run the calculations or logical reasoning myself and conclude the truth of the argument myself. In the case of political science, I can't really do much as I won't be able to recreate such studies.

Answer 1

Three tests that might help:

• Is the source published in a peer-reviewed venue?
• Do multiple independent sources say the same thing?
• If the source cites earlier publications that you've already read, does it tell the truth about what was in those earlier publications?

The more "yes" answers, the more trustworthy the source is likely to be.

Answer 2

A lot of data used in political science is publicly available, so you can replicate what the authors did. That is how you know what studies to trust. I'm not telling you to replicate all studies you read. That would be too much work, especially because you are an outsider in this field. However, you should look at whether you could replicate that study, and if you could do so, then that should increase your trust.

Answer 3

Reproducibility of research in the social sciences is an ongoing and unresolved issue. The problem is our current model of 'good' research is derived from the hard sciences but often the axioms that apply in a field like physics or chemistry are just not suitable in the social sciences. Primarily because social sciences aren't experimental. A physicist can describe in detail the experiment they used to produce a result and then another researcher can follow the same steps and see if the find the same outcome. In a field like political science, you can't go out and run another national election just to test results published by someone else. Consequently, to evaluate high quality work, you're going to have to put some effort into understanding the statistical tools and methods used by the researchers and think critically about whether the published analysis really supports the conclusion.

Some things I might recommend looking for:

1. What is the sample size of the analysis?
2. Are the authors trained political scientists, economists etc. or are they from some other field trying to prove their own pet hypothesis?
3. What is the quality of the references used? Are they citing known journals/work in the field or are they citing 'freemdomeagle.biz'?
4. Are the data used in the research available in an online repository? Most high quality journals these days have a requirement that data sets be made publicly available on platforms like ICPSR, CKAN, Dataverse etc.
5. Are the statistical methods they used appropriate and do they make an effort to explain any statistical anomalies in the data or the results?
6. Have they published the code they used to produce the results? Sadly, this is less common than publishing data but it is a good sign if it is available.
7. Is there a statement that addresses any author conflicts arising from affiliation or funding?

That's what I can think of off the top of my head. Maybe others can add more in the comments?

This is of course all focused on quantitative research. If you want to evaluate qualitative research, that's a whole other question