12

According to this article, a student was arrested for downloading ~4 million articles from an on-line database.

I want to review all existing articles in my interest area before writing my paper, so I need to download and skim through several thousand articles. I read the terms of service of my school's database, but it just says I should avoid trying to make a "collection".

At what point does the downloading become enough that the databases start to complain? 1,000? 10,000? 100,000? 1,000,000? Do they send out warnings?

Improve this question
9
  • 5
    His case was different. He downloaded them to distribute them. That's not fair use. Your case is.
    – Jonathan E. Landrum
    May 13, 2014 at 13:06
  • 15
    Do you really want to skim through thousands of articles? 5,000 articles at 2 minutes each comes to 167 hours, four weeks of single-minded article skimming with no distractions or other responsibilities. I would assume that a more targeted approach (starting with reviews or more recent papers, going through references) might be less mind-numbing.
    – Stephan Kolassa
    May 13, 2014 at 13:10
  • 3
    I suggest you ask for a clarification on "... avoid trying to make a collection". Any half-decent scientist will tell you that you need to have a collection of articles, in order to keep yourself up-to-date, and be able to backtrack and refer to prior work.
    – posdef
    May 13, 2014 at 13:29
  • 9
    also note the following keywords from the article linked above (emphasis mine): "The indictment alleges that between September 24, 2010, and January 6, 2011, Swartz contrived to break into a restricted computer wiring closet in a basement at MIT and to access MIT’s network without authorization from a computer switch within that closet." Assuming that you are using a normal university computer, with proper authorization (which you should have as a scientist there) this article has no real relevance to you.
    – posdef
    May 13, 2014 at 13:33
  • 4
    FYI, the guy mentioned in the article, Aaron Swartz, later committed suicide during the prosecution. He was an internet pioneer and activist; among other things, he was the co-creator of Markdown, the markup language used (also) on Stack Exchange posts.
    – Federico Poloni
    May 13, 2014 at 17:39

2 Answers 2

Reset to default
14

According to this article, a student was arrested for downloading ~4 million articles from an on-line database.

That was an extreme case. He had history of downloading and releasing large amount of information. In this JSTOR case, I don't think he had actually released the articles, although the indication was strong. He was charged with "unlawfully breaking in a protected computer," which I guess he probably downloaded the paper with unusual means, perhaps using codes or hacks. Overall, the whole case was very politically charged and I will not use that as a benchmark in your case.

I want to review all existing articles in my interest area before writing my paper, so I need to download and skim through several thousand articles. I read the terms of service of my school's database, but it just says I should avoid trying to make a "collection".

When there are several thousands of articles that are pertinent to your research interest in ONE paper, I would suggest you to refine your research interest. And no, you don't need to review all existing articles, just pick the important seminal works, and then trace the major works along a very well define research interest. You can also use a few prominent modern works as seeds and use their reference list to snow ball your library.

At what point does the downloading become enough that the databases start to complain? 1,000? 10,000? 100,000? 1,000,000? Do they send out warnings?

You should be able to download as much as you want as long as you follow the instructions and rules. For large download volume, I'd check with the librarian to clarify:

  1. What the institutional limit is. There may be a download limit and if you'd get a warning letter it's likely from your school because your thousands of download could have cost them a lot more than usual. Some school libraries may ban accounts associated with a large download volume as well; it's better to give the librarian a heads up.

  2. Your student account payment scheme. Some schools may have limit and beyond which the students may have to pay. I know that this is certainly possible for inter-library loan (which you may have to do if a few thousands paper is your goal.) A peer of mine got billed for more than US$500 because she ordered about a hundred papers through inter-library loan.

Improve this answer
1

I would suggest to contact the database team, tell them about the project, explain that are you doing, so they would have chance to say is it ok for them or not. You are much less likely to face restrictions if the project is known and approved by your scientific supervisor.

The database website should contain the terms of use page where these acceptable limits should be listed. If nothing is listed, usually a good rule is to wait three times the duration it took a server to serve the previous query but no less than one second, and stop immediately if the server returned the unexpected error message.

It may be much more important not to violate other terms of use.

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .