10

I'm in the process of conducting a systematic literature review and an integral part of this is the definition of inclusion and exclusion criteria for filtering the search-result-publications.

A common criteria for these reviews is that the publication is an academic publication or otherwise academically acceptable. I haven't been able to find a proper definition for this. The ones that I can think of are that the publication fora must exercise peer-review on the papers and that the fora are widely accepted by (inter)national funding agencies.

Hence, my question is what makes a publication or a publishing forum academically acceptable?

Improve this question
2
  • 1
    Where do you conduct your search? If you only do it on SCOPUS or Web of Science, you should already be partway there, since these already pre-filter academic publications. (And you will still need to read the publications themselves and judge their quality.)
    – Stephan Kolassa
    Jan 12, 2015 at 11:24
  • @StephanKolassa we work on a rather new software engineering concept and as such we'd like to consider all possible databases in order to capture all research in the area. For us this means expanding from SCOPUS and WoS to e.g. Engineering Village and even Google Scholar
    – J3lly
    Jan 12, 2015 at 11:38

2 Answers 2

Reset to default
5

Acceptability is in the eye of the beholder. You cannot get this "right" for two reasons:

  1. Different scientific communities have conflicting opinions about about the credibility of various publication venues.
  2. Even "good" venues publish rubbish papers.
  3. Some important papers are published as white papers or technical reports, and will not appear on any list of journals.

To my eyes, the real question is not about what the "right" value is, but about how the boundary that you draw will affect the conclusions of your systematic review.

For example, if you are attempting to perform a meta-analysis on the data within other data sets, then you just need a wide enough scope to be sure to get good statistical validity. Therefore, if your topic is well-studied, you can probably restrict yourself to only those publications listed in some field-appropriate major database, e.g., PubMed for biomedical literature, DBLP for computer science. It doesn't really matter which one, because you're not actually going for comprehensiveness, just for sufficient sampling, and it's more important to get well-curated data than all data. Moreover, the bad publications in the dataset are expected to be drowned out be the good ones in your data processing.

On the other hand, if you are attempting to summarize all of the credible thinking regarding a topic, then you would want to set a much broader criteria, e.g., any journal or conference with at least 5 years of publication history and not on Beale's list. In this case, you can be so generous because you are going to be using a lot more personal discretion in deciding how much weight to give each paper and interesting thinking may turn up in obscure places.

These are the two main cases that I typically see for systematic review; for other cases, you may need to adjust or pick other strategies. In all cases, however, the guiding principle is a) there is no "right" answer, and b) your choice should be driven by the effects it will have on your review.

Improve this answer
5
  • @jakebeal while I now understand that selection of these criteria is very context and research goal specific, I think rules like "5 years of active publication history" and "not on Beale's list" are very good general candidates. Thank you
    – J3lly
    Jan 13, 2015 at 8:47
  • @Davidmh Right you are - I've incorporated this point into the answer.
    – jakebeal
    Jan 13, 2015 at 12:43
  • 2a. ...and even "rubbish" venues publish good papers.
    – JeffE
    Oct 29, 2015 at 13:57
  • The link to Beale's list is not working anymore, is there another alternative somewhere? (and thank you for your great answer!)
    – curious
    Jul 2, 2017 at 3:40
  • @Emilie Unfortunately, I have not yet found a good replacement.
    – jakebeal
    Jul 2, 2017 at 16:05
7

I am afraid this is an impossible question to get right: An "acceptable academic publishing forum" would be one that is accepted by "acceptable academics" - and "acceptable academics" would be identified by publishing in "acceptable academic publishing fora"...

Peer-reviewed is not a bad criterion, but it falls short: There are peer-reviewed journals for creationists and homeopaths - so the value of peer-review depends crucially on who the peers are. It doesn't even work as a necessary criterion, as missing the arXiv for math-y fields or books for humanities would mean that a literature review is very incomplete.

"Acceptance by funding bodies" gets the question backwards: Funding bodies try to approximate what the academic community considers good publication venues, they don't define it.

If formal inclusion/exclusion criteria are important for you (maybe because you do some meta-statistics?), you will need to look for field-specific standards. In medical subjects, I could imagine that "listed on pubmed" would be an acceptable one. In this case, looking are published meta-studies and their criteria would be useful.

Improve this answer
2
  • 1
    In relation to peer-review, there is also the difference of stating that peer-review is conducted and actually doing proper peer-review. Most fake, predatory journals claim that they do peer-review. So this criteria might be clear-cut in theory but harder to apply in practice (especially during a literature search)
    – fileunderwater
    Jan 12, 2015 at 12:41
  • so it is a question of what are the generally applied and hence accepted criterion for one's specific field of research?
    – J3lly
    Jan 13, 2015 at 8:52

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .