Is searching in article references considered a systematic way of literature search?

Question

Doing a systematic review, either quantitative or qualitative, requires developing a well-defined protocol as a method of conducting it. This includes defining the search methods used in the identification of eligible articles, which include two methods:

1. database search using search queries (a main method)
2. searching in the references and cited-by sections of the selected articles (which are already retrieved by the first method)

Let's assume that an author (of a large review) does their best to formulate the search queries to better represent the review question, but later they realized that considerable amount (e.g. 50% or more) of the relevant articles are found only by using the second method (searching in articles);

Is the second method less 'systematic' than the first one? Does it compromise how much 'systematic' the resulting review is?

If so, does the above scenario affect the validity of the search queries used (i.e. should they be reformulated to increase the recall of search and retrieve more of relevant articles)?

Edit: More extremely: If 90 out of selected 100 articles are found only by using the second method, how much does this affect the quality of the systematic review?

Answer 1

I would add to the other answer(s) that you could validate the first approach using the second approach. If your search criteria systematically miss relevant articles in the journals covered by the database, then that's a clear sign you need to reformulate your search criteria because they're clearly not sufficient.

The second thing to add is that any search should also try to find unpublished articles. These are definitely not going to be in databases, but may be cited in papers (thus the importance, at least in my view, of the second approach). It might be worth adding a third approach that involves search conference abstracts, where unpublished (especially recent) work is likely to be found.

Answer 2

Databases are not complete and published articles do not necessarily reference all pertinent literature so it seems unlikely that one would necessarily capture everything relevant by using just one method.

Databases are probably relatively complete when regarding more recent publications. I would not like to define "recent" however but I see it as mostly post-1990's. But my guess is that databases are centered around more widespread journals and more local journals may not be well-represented. This means that depending on the search area they may be more or less complete. Going back in time more and more will likely not be found in databases so if the topic has a vital history then database searches will cover the "recent".

Reference lists may pick up more older material but that is of course dependent on the authors willingness to research literature. It is possible one might pick up more esoteric references this way but I fear the selection will be fairly random and not comprehensive.

So to use both methods seem like the safest way forward to me. Depending on the subject matter, having deeper understanding for where and when things might have been published in the past may be vital in order to capture most relevant literature on the subject. To venture so far as to say everything will be found is difficult. Particularly during the cold war much was published in for example Russian journals that never reached the west. Many discoveries published in the west were therefore missing out on the eastern counterparts and may not even have been first. Much has thus been lost by lacking translation and that goes for many if not most languages. One must also remember that the publication scene as we see it today was not utilized earlier, when internal reports and local journals may have taken up much research. To rely on just one of the two parts for methodology may therefore be inadequate.