4

I am doing research in text mining. For that I need a dataset which contain full text PDFs of research paper articles, and all articles should be related with each other in terms of citing. There are citation network datasets available which contain only metadata of papers, but with this I also want those full text articles in PDF format.

What should I do to get such dataset?

migrated from cs.stackexchange.com Sep 24 '14 at 7:31

This question came from our site for students, researchers and practitioners of computer science.

  • Google Scholar, Mendeley and probably other tools already know who cites whom. Have you tried extracting data from such? – Raphael Sep 24 '14 at 7:33
  • 4
    You are most likely required to create such a data set itself as papers are copyrighted - thus unless you restrict yourself to CCed papers, nobody could provide you with such a data set without violating copyright laws. – DCTLib Sep 24 '14 at 8:35
  • Your question is completely unclear to me. – Enthusiastic Engineer Sep 24 '14 at 10:04
  • @DCTLib: Not so fast. See my answer. – Nate Eldredge Sep 24 '14 at 14:36
  • I'm not totally clear what you mean by "all articles should be related with each other in terms of citing." Can you clarify? – Nate Eldredge Sep 24 '14 at 14:36
2

An alternative to arXiv mentioned in another response could be PubMed Central, specially if you are interested on text mining of papers in biomedical and life sciences research.

1

My first thought would be arXiv, where you can find 973,256 full-text papers.

You may not bulk-download them directly from arXiv.org, but other methods are available. In particular, you can download every PDF hosted by arXiv via Amazon S3, by paying a bandwidth charge of USD 0.12 per gigabyte. See these instructions. Metadata in various formats is also available.

Contrary to DTCLib's assertion that "nobody could provide you with such a data set without violating copyright laws", it is perfectly legal for arXiv to do this, since each author submitting a paper to the repository gives arXiv the right to redistribute the paper to anyone. Note, however, that does not in general give you the right to redistribute them further. (Some papers on arXiv are released under Creative Commons licenses, or even are in the public domain, but most are not.)

  • I see. I was assuming that the OP needs the citation graph along with the papers (it is actually not precisely stated in the question whether she/he does). While many papers offer the TeX source code along with the paper, which could be used to extract it, by far not all do. Perhaps the OP could match the publicly available citation graph against the ArXiv papers? – DCTLib Sep 25 '14 at 8:04
1

ACL Anthology Network is a dataset, that contains about 22,000 articles related to conferences under the umbrella of ACL. they deal with papers in coputational lingusitics. The dataset contains text files for all the papers from 1960s to 2014. Some network information is also provided in the dataset.

http://clair.eecs.umich.edu/aan/index.php

If you want more structured XML files, the same can be obtained, but needs to be downloaded with a mass downlaoder, and is available for only about 9500 of papers. Also individual PDFs also need to be obtained separately, if required

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.