4

I am doing research in text mining. For that I need a dataset which contain full text PDFs of research paper articles, and all articles should be related with each other in terms of citing. There are citation network datasets available which contain only metadata of papers, but with this I also want those full text articles in PDF format.

What should I do to get such dataset?

Improve this question
8
  • Google Scholar, Mendeley and probably other tools already know who cites whom. Have you tried extracting data from such?
    – Raphael
    Sep 24, 2014 at 7:33
  • 4
    You are most likely required to create such a data set itself as papers are copyrighted - thus unless you restrict yourself to CCed papers, nobody could provide you with such a data set without violating copyright laws.
    – DCTLib
    Sep 24, 2014 at 8:35
  • Your question is completely unclear to me.
    – enthu
    Sep 24, 2014 at 10:04
  • @DCTLib: Not so fast. See my answer.
    – Nate Eldredge
    Sep 24, 2014 at 14:36
  • I'm not totally clear what you mean by "all articles should be related with each other in terms of citing." Can you clarify?
    – Nate Eldredge
    Sep 24, 2014 at 14:36

3 Answers 3

Reset to default
2

An alternative to arXiv mentioned in another response could be PubMed Central, specially if you are interested on text mining of papers in biomedical and life sciences research.

Improve this answer
1

My first thought would be arXiv, where you can find 973,256 full-text papers.

You may not bulk-download them directly from arXiv.org, but other methods are available. In particular, you can download every PDF hosted by arXiv via Amazon S3, by paying a bandwidth charge of USD 0.12 per gigabyte. See these instructions. Metadata in various formats is also available.

Contrary to DTCLib's assertion that "nobody could provide you with such a data set without violating copyright laws", it is perfectly legal for arXiv to do this, since each author submitting a paper to the repository gives arXiv the right to redistribute the paper to anyone. Note, however, that does not in general give you the right to redistribute them further. (Some papers on arXiv are released under Creative Commons licenses, or even are in the public domain, but most are not.)

Improve this answer
1
  • I see. I was assuming that the OP needs the citation graph along with the papers (it is actually not precisely stated in the question whether she/he does). While many papers offer the TeX source code along with the paper, which could be used to extract it, by far not all do. Perhaps the OP could match the publicly available citation graph against the ArXiv papers?
    – DCTLib
    Sep 25, 2014 at 8:04
1

ACL Anthology Network is a dataset, that contains about 22,000 articles related to conferences under the umbrella of ACL. they deal with papers in coputational lingusitics. The dataset contains text files for all the papers from 1960s to 2014. Some network information is also provided in the dataset.

http://clair.eecs.umich.edu/aan/index.php

If you want more structured XML files, the same can be obtained, but needs to be downloaded with a mass downlaoder, and is available for only about 9500 of papers. Also individual PDFs also need to be obtained separately, if required

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .