Using public profile data from website for research

Question

A member of our department plans to download millions of profiles from a public website. The data is freely accessible after you have logged in, though there is a company behind the website that does have commercial interests and sells premium memberships. The website’s terms and conditions don’t explicitly prohibit mass downloading of information and he claims that his software would simulate normal user behavior, thus not interfering with the normal operations of the site.

A preliminary evaluation from our ethics committee suggests that – from an ethical standpoint – everything would be fine as long as he doesn’t publish the dataset (which he doesn’t plan to do) and the data is anonymized. Also, he will aggregate the data, so no information pertaining to individuals will be intelligible.

Still, I am not completely convinced that this study should be done and the final vote is still pending. Are there any other things to consider? Could the company operating the website possibly sue the university (or him) for using the data or would that be condoned by an act protecting the freedom of press or anything similar? We are in Germany, so German and EU law applies.

Answer 1

What you're describing is the OKCupid data release, just without publishing the dataset. Note that the OKCupid "study" and similar studies on Facebook data were considered ethically controversial by some, who argue that users who participate in a website are not necessarily giving consent to participate in a research study, especially when users' profile data is only available to logged-in users. (Also see: “But the data is already public”: on the ethics of research in Facebook by Michael Zimmer.)

Re: avoiding a lawsuit, you should get the OK from the website. As Wrzlprmft pointed out in a comment:

Unless the research could cast them in a bad light, they may be interested to learn something about their userbase as well. Also, you may get to know relevant information about the data, such as hidden selection mechanisms or similar

If you choose not to go that route, I'd recommend consulting with a lawyer re: the website's terms and conditions. Realistically, nobody here can tell you the likelihood of being sued, or of losing the lawsuit. A lawsuit would probably be based on the terms and conditions of the website; get a lawyer to review them. Your colleague's assessment that they "don’t explicitly prohibit mass downloading of information" is not exactly reassuring. (For example, language in the terms and conditions that enumerates the permitted use of the website can potentially exclude other uses, without explicitly enumerating the prohibited uses.)

The case law on this issue is far from settled. The decisions so far have been limited. There is no legal precedent that clearly establishes that "scraping is always allowed if the T&C don't explicitly prohibit it". There are some select cases that have begun to map out cases where scraping is and is not prohibited, but the legal territory is not yet fully explored. Some further reading on case law in the US and EU:

• Judge: Google cache kosher when it comes to copyright
• Judgment in Ryanair Ltd v PR Aviation BV
• Judgment in Innoweb BV v Wegener
• Web Scraping in an Era of Big Data 2.0