Ethics of using company data from internship for academic project

Question

I am a graduate student and one of my research projects requires me to crawl a website using their public API. This is going alright but the amount of data we can collect is limited, since it is a public API.

As it happens, I will be doing an internship at the company that owns that website, and will soon have access to an unlimited supply of their data. Is it okay if I use that data for my academic research project, or will they expect me to only use their data for projects relating to my internship?

Answer 1

You should ask the company where you do your internship. Perhaps they are fine with whatever you want to do. Perhaps they will allow you to use their data but only release summaries, not the raw data. Perhaps they will require you to anonymize it.

In addition, how will you do the analysis? Will you take the company's data and transfer it to your own computer? (This may well not be welcomed by the company.) Or do you plan on using a company-supplied machine to do your analyses? (Same problem - companies don't necessarily like it if you use their hardware for what are essentially personal tasks.) Do you want to install third party software to analyze your data on their machines?

Think through what you want to do, then go to your supervisor. Depending on their personality, people may even be interested in working with you on this, perhaps get a publication out of it.

Whatever you do, don't, don't, don't just use the data without asking. Whether or not you sign an NDA.

Answer 2

Ask them and see what they have to say. It's one thing to crawl their public api in your own time ( which actually may specifically prevent you from doing your research according to its terms of use ) but using their internal data directly without asking would be unethical at best and potentially illegal at worst. ( and the last thing that you would want is to be known as the guy that stole people's personal information for their own gain then got dragged through court when they made it public) while your research may only contain aggregate data or anonymised datasets then that still won't change the perception of potential future employers.

Answer 3

• You have to ask for permission, in writing [*]. Don't just use the data without asking, that would be terminally unethical. If they are reluctant or have a confidentiality or security concern, you (and/or your research supervisor) need to make the case to them in terms of how they will benefit from publications and exposure - for reputation, recruiting, etc. Conduct this negotiation at the highest appropriate level you can manage, i.e. involve the people who have the authority to say yes and are actually incentivized to. (Also good for your career recognition). Sometimes you need the engineering dept to nudge the legal dept on your behalf, or override stupid policies.
• Define whether you can/should anonymize/double-anonymize/ aggregate/sample/fuzz the data (e.g. look at how the Census does it). Or maybe only use non-identifying fields. They may ask you to take reasonable efforts to secure it (password, encryption, firewall, external HDD), but don't agree to unlimited indemnification.
• Try to get the agreement as broad as possible, i.e. no time-limits, no limits on (non-commercial) scope, allow derivative works by you, your dept or your co-researchers. Agreements will almost always forbid derivative commercial use. If you come up with a patentable idea, who holds the rights? etc.

[*] UPDATE: nightmares like the following (a retrospective IP policy) are precisely the sort of thing that can happen: "PhD student, issued contract at year 3 which will sign over intellectual property. Is it legal?". Now that person can have all their publications vetoed and their thesis closed by the industrial partner.

Answer 4

I will be doing an internship at the company that owns that website, and will soon have access to an unlimited supply of their data. Is it okay if I use that data for my academic research project

NO!

No with the same strength as the answer to the question, "I'm doing research on wages in different professions. A friend has invited me round for dinner next week. Is it OK if I rifle through his filing cabinet to see if I can find a payslip?"

If you want to use somebody's data and that data is not unambiguously public, you get their explicit permission, first. In a corporate environment, there are many reasons why permission may be refused: for example, the data may be commercially sensitive or the company might have privacy obligations towards its customers. The fact that you need privileged access to get hold of the data should tell you immediately that you need permission to use it for anything other than the specific reason for which you were granted that access.

Answer 5

What you are talking about doing is technically considered corporate espionage without explicit permission from the client.

Facts (as you have stated them):

1. You are performing academic research consuming public data of Corporation A
2. You will be taking on an internship with Corp A and have access to Corp A's private information
3. Your capacity within Corp A is disjoint from your academic research involving Corp A.
4. You are at least entertaining the idea of using Corp A's private data for research purposes without their consent.

So my recommendation is entirely dependent upon the nature of the relationship between your internship with Corporation A and their knowledge of your research project involving their data.

If they knew about your research project before granting you an internship, I see it as entirely appropriate to ask them for permission.

If they did not know about your research project before granting you an internship, you put your internship at risk by asking for permission to use their private data, depending on the sensitivity of the data you would have access to and the personalities of the business members in control of your relationship with the corporation. The risks associated with you violating their denial of your request could be more trouble than you are worth.

If they did not know about your research project before granting you an internship and you publish private corporate data without explicit consent, they have a legitimate claim that you sought your internship with them for the express purpose of gaining dubious access to their private information and you would be in for a world of hurt both legally and professionally. Depending on the fallout of that, you could reasonably expect to find yourself to never be employable again in your field and you could reasonably expect to find yourself in prison depending on the nature of the data that you expose and the political clout that is held by the people that you piss off.

Edit To Add

You may already be between Scylla and Charybdis on this. Look at the potential optics: you have a research project (the nature of which you have not disclosed.) You also have an internship with access to corporate private data of one of the subjects of your research study. Assuming that you do not announce your research project and you take the internship, your standing both within your academic community and within the corporate world will definitely hinge on the nature and results of your research project.

If you take an internship with them and then crucify them in your research project, their quid pro quo will be reciprocal regardless of whether or not they consent to you publishing their private data. With access to their private data, your research methodology will be called into question (perhaps legally) and any potential mistake you may have made will be exacerbated. If your results are flattering to Corporation A, then your academic peers may consider you to be politically beholden to Corporation A.

You may already be in a no win scenario other than to cancel your research project or cancel your internship. You really are not supposed to do what you are thinking about doing.