For example PhD scholars working on ML/AI algorithms need a lot of data to train and test their research. Research on distributed systems might require terabytes of data to test.

Where do these scholars get so much data i.e. the scale which is only produced in big companies?

  • 1
    They create relationships with companies to be able access the data or they purchase (buy, pay for) it... – Solar Mike Nov 13 '18 at 14:50
  • If you work in the biological sciences there are lots of databases: national center for biotechnology information, protein data bank, national institutes of standards and technology. – xyz123 Nov 13 '18 at 16:53

The question is very vague. For some fields such as linguistics/language there are huge open-access research corpuses available such as the British National Corpus, maintained by academic or charitable organisations. For some things, e.g. images, you can also use open, permissively-licenced sources like Wikimedia Commons, which offers a lot of APIs for searching and processing; the Internet Archive has a huge collection of video, texts, audio, etc. For some subjects it is possible to use internet searches or other archives. Many commercial entities will have large databases and research project are often done in conjunction with a commercial partner, as mentioned in a comment. You can also purchase data, but that will probably require a commercial partner to pay for it.


One of the things you should remember is that nothing in life is free, especially not data. I have seen instances of inexperienced students/postdocs reaching out to companies asking for their data (without informing their advisors). These requests go either ignored, or politely refused. The company needs to know that there is some real benefit in working with you, whether it is an improvement to their current processes or a monetary incentive. In either case, it often involves the university's lawyers making sure that the transaction is

  1. Legal/ethical - the data you're getting doesn't require someone's consent and is indeed owned by the company. You may need to get IRB involved too.
  2. Transparent for both sides - what are you going to do with your analysis? Will the data be published? Will it be anonymized? Can you patent the results? If so - what part of the patent is owned by the university/company/yourself?
  3. Modes of compensation for the company, if any.

(this is a partial list, I'm not a lawyer)

This is even less fun when one works on data provided by a government agency. This is why often enough researchers either work on public datasets (e.g. those on Kaggle/Github), collect their own, run simulations, or just go on research visits to Amazon/Uber/Google/Facebook to get access to actual data, see that their models make sense, and get a nice paycheck while doing so.

Not the answer you're looking for? Browse other questions tagged or ask your own question.