What are acceptable criteria to select publications for a (systematic) review?

Question

I'm in the very beginning of my scientific career, so I don't yet know the standards for (systematic) reviews very well. I feel I don't have so much experience that I could publish my own research in good journals (I have now just two conference papers), so I'm considering to pay close attention for 6-12 months in other peoples' works.

I have a clear research question, but given the massive amount of related papers it's difficult to choose which ones to pick. Of course, I will select papers related to my research question as the first criteria. However, could I narrow the range of included papers more by

• publication date (e.g. only take papers published during the past five years),
• impact factor or citation count,
• type of publication (conference, journal),
• sample size (e.g. number of subjects)?

Or, if not these, what criteria are commonly used? I guess the answer is different for reviews and systematic reviews, so please cover both if possible. thank you!

EDIT: I'm considering this so that it would be possible to try publish my results.

Answer 1

I do not find any of these points very relevant. The publication date couldbe useful if your topic is such that older references are indeeed obsolete.

To write a review you want to collect papers that describe your topic as well as possible. A review is usually a good point for others to see the depth and breadth of a subject so being complete is usually a sign of quality.

Given that you have a topic, you need to consider how you should organize your information. Is there any new that can come out of the way you organize your review? A review is not just a collection of older results, you need to provide a synthesis. Often the outcome is to made limitations in older work visible or to identify gaps in knowledge or point at directions for new development. Depending on what the purpose will be you may end up chosing papers differently. However, often you do not know the structure until you have read enough and come to realize how the knowledge in th earea is distributed.

So for me there is no real difference between review and systematic review. A review is simply a way to orgnaize and sort older information in a new way so that new patterns emerge and hint at directions for further research or other forms of new ideas.

Answer 2

In health related fields there is a huge difference between a review and a systematic review. The point of a standard review is to summarise an area of research possibly identifying areas of future research, but not necessarily. A standard review is often conducted in conjunction with a meta analysis or leads to a meta analysis to determine if a particular Patient population is helped by an Intervention more than a Comparison intervention on a set of Outcomes. This leads to the PICO search strategy. There are a number of different frameworks by which systematic reviews can be conducted. A Cochrane review is one such system. There is a whole field of literature associated with conducting systematic reviews.

The goal of a systematic review, and a review in general, is to find all the relevant papers. If you want less papers you need to narrow your question. PICO can help with this.

Answer 3

The point of a review of a field or subfield (IMHO) is to tell the story of the development of the field, leading up to explaining what are the major open questions? and why are they important? Often (though not always), you should focus on big ideas and which developments they have lead to. If there was a sequence of 10 papers that repeatedly applied roughly the same ideas to larger data sets, because they got bigger and faster computers, that is probably not very interesting, and something you will likely gloss over (or completely omit) from your review. So you need a way to determine which are the most important papers and start with those.

It's hard to write a review before you know the field. Here are a few possible ways to get that knowledge quickly:

• Read an old review in the same area. See which papers it claims are foundational, then check to see which more recent papers cite the foundational ones. Iterate.

• Go to lots of seminar talks and see which results always get mentioned in the introduction. Start by reading those papers.

• Ask a more senior student or faculty member where to start.