4

A member of our department plans to download millions of profiles from a public website. The data is freely accessible after you have logged in, though there is a company behind the website that does have commercial interests and sells premium memberships. The website’s terms and conditions don’t explicitly prohibit mass downloading of information and he claims that his software would simulate normal user behavior, thus not interfering with the normal operations of the site.

A preliminary evaluation from our ethics committee suggests that – from an ethical standpoint – everything would be fine as long as he doesn’t publish the dataset (which he doesn’t plan to do) and the data is anonymized. Also, he will aggregate the data, so no information pertaining to individuals will be intelligible.

Still, I am not completely convinced that this study should be done and the final vote is still pending. Are there any other things to consider? Could the company operating the website possibly sue the university (or him) for using the data or would that be condoned by an act protecting the freedom of press or anything similar? We are in Germany, so German and EU law applies.

  • 2
    What you're describing is the OKCupid data release, just without publishing the dataset. I'd recommend consulting with a lawyer re: the website's terms and conditions. Or getting an OK from the website. – ff524 Dec 27 '16 at 9:04
  • 1
    I strongly suggest talking to the company owning the website. Unless the research could cast them in a bad light, they may be interested to learn something about their userbase as well. Also, you may get to know relevant information about the data, such as hidden selection mechanisms or similar. – Wrzlprmft Dec 27 '16 at 9:07
  • (Realistically, nobody here can tell you the likelihood of being sued, or of losing the lawsuit. A lawsuit would probably be based on the terms and conditions of the website; get a lawyer to review them. Your colleague's assessment that they "don’t explicitly prohibit mass downloading of information" is not exactly reassuring.) – ff524 Dec 27 '16 at 9:16
  • How does Google actually hold a lot of websites in cache if they do not download them initially - would they violate all websites ? Regardless of T&C ? On the other hand, I have heard multiple times that it is prohibited to scrape any kind of website in Germany without the OK of the website owner. – Michael Dec 27 '16 at 9:59
  • 1
    @Michael Further reading on that lawsuit, in case you're interested. – ff524 Dec 27 '16 at 10:17
2

What you're describing is the OKCupid data release, just without publishing the dataset. Note that the OKCupid "study" and similar studies on Facebook data were considered ethically controversial by some, who argue that users who participate in a website are not necessarily giving consent to participate in a research study, especially when users' profile data is only available to logged-in users. (Also see: “But the data is already public”: on the ethics of research in Facebook by Michael Zimmer.)

Re: avoiding a lawsuit, you should get the OK from the website. As Wrzlprmft pointed out in a comment:

Unless the research could cast them in a bad light, they may be interested to learn something about their userbase as well. Also, you may get to know relevant information about the data, such as hidden selection mechanisms or similar

If you choose not to go that route, I'd recommend consulting with a lawyer re: the website's terms and conditions. Realistically, nobody here can tell you the likelihood of being sued, or of losing the lawsuit. A lawsuit would probably be based on the terms and conditions of the website; get a lawyer to review them. Your colleague's assessment that they "don’t explicitly prohibit mass downloading of information" is not exactly reassuring. (For example, language in the terms and conditions that enumerates the permitted use of the website can potentially exclude other uses, without explicitly enumerating the prohibited uses.)

The case law on this issue is far from settled. The decisions so far have been limited. There is no legal precedent that clearly establishes that "scraping is always allowed if the T&C don't explicitly prohibit it". There are some select cases that have begun to map out cases where scraping is and is not prohibited, but the legal territory is not yet fully explored. Some further reading on case law in the US and EU:

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.