4

I'm a researcher, and I want to scrape a website (specifically, TripAdvisor) to collect data in order to use it for a research project. However, in reading the website's terms & conditions, I found that the company prohibits the user from:

...copy any content or information of this Website using any robot, spider, scraper or other automated means or any manual process for any purpose without our express written permission.

-- Prohibited activities, bullet (ii)

As such, I want to contact the company to ask for permission. I've had a difficult time finding any information on how to contact the company in order to ask for permission to use data. That said, my question is, when making these sorts of requests to companies, what is the best way to ensure that I am contacting the right department/person?

Improve this question
15
  • Did you start with they're contact information? tripadvisor.com/PressCenter-c6-About_Us.html
    – user60356
    Aug 16, 2016 at 15:54
  • 1
    This is both too localized and not on topic. At best, you could edit to say "how to approach online aggregators to use their data for research", but I suspect that question would only have answers that don't help you ("find the email and email them"). FYI tripadvisor has a whole help center website where your efforts are probably better spent.
    – Superbest
    Aug 16, 2016 at 17:11
  • 1
    @Superbest - I've made a few edits to make the question more general. I think there's a good question to be asked here that's probably relevant to quite a few researching hoping to get data off public websites with similarly restrictive TOS.
    – eykanal
    Aug 17, 2016 at 16:52
  • 1
    @MOHBOB: "I foud only their physical mail adress. I need their e-mail." - well ... why? Why not simply send a letter (yes, on paper)? Deleting an e-mail is still a little bit easier to do than "erasing" a physical letter.
    – O. R. Mapper
    Aug 17, 2016 at 18:39
  • 1
    And, just for the sake of completeness, here is a forum thread on tripadvisor.com where another researcher wanted to get some data access in April 2016.
    – O. R. Mapper
    Aug 17, 2016 at 18:41

3 Answers 3

Reset to default
2

Firstly, recognize that they probably placed that ToS there for some or all of the following reasons:

  1. They want a legal basis for suing someone who tries to steal stuff from their site.
  2. They want a legal basis for suing some idiot programmer who brings down their site because of a poorly programmed bot he wrote "for a project".
  3. Their lawyer told them to put these ToS up because he saw them on some other large site and thought they looked good there.
  4. ...?

You likely fall in the second category (and maybe the fourth, I don't know what other things they're protecting themselves from). To that extent, you have to convince them that you aren't an idiot who will run amok on their servers. If you're aligned with a well-known university then maybe you'll have more luck, but even then it'll be a stretch... this is a for-profit company, not a research institution. Even if you can convince them that you're not a total idiot, they still have very little reason to want to grant you this permission. Your pet project will gain them nothing, and may in fact demonstrate something bad about their site that they don't want public. The "I'll share my findings" argument really doesn't go very far... companies have their own interests, governed by many things, including politics, agendas, egos, and what have you. Your powerpoint presentation that may be done in a year or two that will probably be too technical for a business VP to understand anyways won't give them any business value.

The point of all the preceding words is just to convince you that you have quite the uphill battle here.

All that said, I would just start calling every "contact us" number I can get my hands on. Definitely start with the numbers on the "Contact Us" page. Don't be afraid to send a written letter! For phone calls, I like the phrase Jack St Claire used in his comment: "I have a weird question."

Success is easy to determine. Check your mail... do you have a letter (written or electronic) from the TripAdvisor legal team permitting you to perform the research? If no, then you haven't succeeded yet. Keep trying! Perseverance will be key here.

Improve this answer
7
  • "but even then it'll be a stretch" - while your counter-arguments may be true in principle, I still think that statement sounds overly pessimistic. Research projects getting some temporary access to a company's data are by far not a rare occurrence ... or maybe that is my impression, given that I am from a field that is more likely to simplify usage of a company's services than reveal any undesirable details. In any case, my impression is that the companies' primary concern is less that a "pet project will gain them nothing" (that is somewhat compensated for by the simple fact that they'll ...
    – O. R. Mapper
    Aug 17, 2016 at 18:36
  • ... get another little bit of publicity, (mostly) for free), but rather, that some of their data is not meant for public access (user data, paywalled data, ...), and that, once they have agreed to provide data, it can still be a bit of a struggle to actually get something moving.
    – O. R. Mapper
    Aug 17, 2016 at 18:36
  • That is exactly my opinion @O.R.Mapper.
    – Moh_BOB
    Aug 17, 2016 at 18:45
  • @eykanal : I work on recommendation systems, and I see very often academic research projects using data from Tripadvisor. For the argument "won't give them any business value.", it seems that Tripadvisor have the habit to give access to researchers, but request the findings as a counterparty. I think that projects like recommender systems could be very beneficial for their type of service.
    – Moh_BOB
    Aug 17, 2016 at 18:45
  • @MOHBOB - If TripAdvisor has a good reputation here, awesome! That's much to your advantage. From both my experience and that of my colleagues, most companies won't even bother to give you the time of day.
    – eykanal
    Aug 17, 2016 at 19:21
0

I would suggest 2 things.

1) contact other academic researchers who got this data previously (you mention in the comments this has happened) and ask how they did it. This is really your best bet and the only thing that has ever really worked for me.

2) Use LinkedIn to figure out who is likely to handle such a request. For TA it could be someone in their legal department or in their machine learning group, perhaps. You can at least get the sense here of what the departments are. I have used this method in the past to get responses to my requests, although they were "no".

Also I will note that I once had permission from a company to scrape data and then the company did not shut me down, my university did because it thought I was a "hacker from China". Then both legal departments got involved and took the data back. So try to get a dataset directly from the company, not permission to scrape.

Improve this answer
-1

I finally find help on ResearchGate.com. One researcher provided me some email addresses that he founds on this page and suggested that I email a request to the CEO (Steve Kaufer) asking for permission. I've emailed him a request, and I'm waiting his response. Hope this can help anyone in future.

Improve this answer
3
  • 1
    Two comments: (1) The email address you posted is not necessarily correct (e.g., I found another one here... I suspect they're just following common email address formats), and (2) emailing the CEO of a $9B company asking for permission to perform a research study is almost certainly a dead end. The likelihood of your getting a positive response is practically zero.
    – eykanal
    Aug 17, 2016 at 16:46
  • Not trying to discourage you, but definitely trying to encourage paths that are more likely to be productive.
    – eykanal
    Aug 17, 2016 at 19:22
  • This is really not the person you want. You want someone like the head of research or the head of analytics.
    – Dawn
    Aug 19, 2016 at 11:34

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .