Finding papers that report results on a dataset

Question

I am doing a literature study, and I would like to identify published research papers that report results on Reuters dataset RCV1-v2 (as comprehensively as could possibly be expected). The results should preferably be comparable (i.e. use the exact same subset of this dataset, e.g., some tend to use only a smaller subset)

How should one approach this problem?

So far, this is what I did:

The dataset was introduced in this paper: Rcv1: A new benchmark collection for text categorization research.

I went to scholar.google.com, found the paper, clicked on "citations", then marked "search within citation", and used keywords such as "results", "comparison", "evaluation". Is there a better way?

There are 1651 citations (according to Google scholar...), at what point should one feel reasonable comfortable that one has (comprehensively) identified the papers which have reported (comparable) results on this dataset?

The goal of the literature study is to motivate my choice of method (I want to inform myself of what methods have been applied, and how they compare to each other).

Answer 1

In principle, I think your approach is good, but I have a few suggestions to make your search hopefully more feasible:

• Rather than using Google Scholar, I would use a more focused scholarly database that also tracks citations. In general, Web of Science is best for these purposes, but since this particular article is available on the ACM Digital Library, then that might possibly be a better choice. If you're interested in computer science and related literature, then ACM Digital Library is the way to go; if you want to be more broad in your range of disciplines, then I would go with Web of Science.

• There are two advantages to using a more focused database rather than Google scholar:

  1. Practically speaking, you have much fewer citations to deal with, and so the task is more feasible. Yet, these focused databases have fairly stringent quality standards, and so although most citations are not included, those that are are widely recognized as high quality. It is a compromise for the sake of feasibility, without meaningfully sacrificing quality. To me, this works for your particular purpose because surveying methods doesn't require an exhaustive search; a comprehensive search is sufficient.
  2. These focused databases usually have much more fine-tuned search options than Google Scholar (e.g. abstract only, keywords, etc.). Thus, you can more precisely focus your search to exclude the articles that you consider irrelevant. (The reason that Google Scholar is not as fine-tuned obviously has nothing to do with their engineers' capabilities; I guess it's probably a matter of copyright licensing to archive and search the journal databases. Google Scholar is free, whereas the others have to pay the journals, so I guess they place a limit on what they consider "fair use" for Google. This is my speculation; I'm not certain.)

Answer 2

Wikidata has the ability to describe datasets and scholarly articles that use datasets. The annotation has been used sparingly and is far from complete.

An example of a scientific article that annotates with a data set is https://www.wikidata.org/wiki/Q62428109 It records use of MNIST, ImageNet (as well as the programming libraries PyTorch and TensorFlow).

The Wikidata Query Service can be use to query this data across pages. You will see an aggregation of the data in Scholia. Here for MNIST: https://tools.wmflabs.org/scholia/use/Q17069496