Doing a systematic review, either quantitative or qualitative, requires developing a well-defined protocol as a method of conducting it. This includes defining the search methods used in the identification of eligible articles, which include two methods:
- database search using search queries (a main method)
- searching in the references and cited-by sections of the selected articles (which are already retrieved by the first method)
Let's assume that an author (of a large review) does her best to formulate the search queries to better represent the review question, but later she realized that considerable amount (e.g. 50% or more) of the relevant articles are found only by using the second method (searching in articles);
Is the second method less 'systematic' than the first one? Does it compromise how much 'systematic' is the resulting review?
If so, does the above scenario affect the validity of the search queries used (i.e. should they be reformulated to increase the recall of search and retrieve more of relevant articles)?
Edit: More extremely: If 90 out of selected 100 articles are found only by using the second method, how much does this affect the quality of the systematic review?