5

It seems to me that machine learning methods are being adopted more and more often in empirical academic research and modelling. Whereas scientists are often quite familiar with traditional statistical methods to report on their findings, they may by comparison often be somewhat less knowledgeable about the machine learning algorithms they employ.

I wonder whether there are any articles on the (in)correct communication of machine learning algorithms in empirical research. I'm interested in how scientists ought to present the results found by means of such methods, to what extent they should describe the algorithms they've used, and other related matters they should take into consideration.

8
  • Here's an example of what I mean by an article in which machine learning methods are used in empirical research and modelling: arxiv.org/abs/1711.05462
    – Max Muller
    Apr 19, 2020 at 10:44
  • Welcome to Academia SE. If you want resources on how to do machine learning, this is not a good place to ask; the content of research is off-topic here. You may be able to ask such questions on Cross Validated or Data Science. Resources on how to communicate results from machine learning would be on the border of our scope, but you will still probably get better answers elsewhere.
    – Wrzlprmft
    Apr 19, 2020 at 10:55
  • @Wrzlprmft please note that I am not asking for resources on how to do machine learning. I am looking for resources on how to communicate results in any empirical science, which have been obtained by means of machine learning methods. Surely that is within the scope of this SE page?
    – Max Muller
    Apr 19, 2020 at 11:07
  • As I said, it’s borderline. Also, “the (in)correct usage of machine learning algorithms in empirical research” and “drawing conclusions” are not about communication. Please edit your question to make this more clear.
    – Wrzlprmft
    Apr 19, 2020 at 11:17
  • 2
    I think it's a great question (+1) Apr 19, 2020 at 14:09

1 Answer 1

1

Probably, precise communication guidelines will (and should) vary between different fields, depending in part on how well machine learning methods have been described, utilized and benchmarked in prior works within the field. However, in my opinion, every empirical field should already be able to take note of lessons learned within computer science, and also implement sanity checks on results based on both general and field-specific knowledge. Indeed, it's not all that different from applying statistical methods to a given study.

The main new challenge is guaranteeing reproducibility while using ML methods. See e.g. M. Hutson: Artificial intelligence faces reproducibility crisis, Science (2018). (Non-paywall link here.) The article stresses the need of providing code, test data, and details on training procedures, but does not provide very specific guidelines.

However, Joelle Pineau, professor of computer science at McGill, has been pushing for higher reproducibility standards in machine learning-related research. Although not published in the traditional sense, the Reproducibility Checklist on her website is the closest to a standard guideline I know of. For example, it was used for submissions to the 2019 NeurIPS conference. (A report on how this worked out can be found here.) It gets routinely updated, so I recommend checking the source. However, I will reproduce some of the points that are relevant for empirical science:

For all datasets used, check if you include:

  • The relevant statistics, such as number of examples.
  • The details of train / validation / test splits.
  • An explanation of any data that were excluded, and all pre-processing step.
  • A link to a downloadable version of the dataset or simulation environment.
  • For new data collected, a complete description of the data collection process, such as instructions to annotators and methods for quality control.

For all reported experimental results, check if you include:

  • The range of hyper-parameters considered, method to select the best hyper-parameter configuration, and specification of all hyper-parameters used to generate results.
  • The exact number of training and evaluation runs.
  • A clear definition of the specific measure or statistics used to report results
  • A description of results with central tendency (e.g. mean) & variation (e.g. error bars).
  • The average runtime for each result, or estimated energy cost.
  • A description of the computing infrastructure used.

This was based on version 2.0 of the checklist, dated April 7, 2020. Note that experimental above does not mean experiment in the sense of empirical science, but in the sense of running a "numerical experiment" on some set of data.


Finally, note that this is a developing field. I fully expect more guidelines to be written over the next few years, as people realize the need for it. For a sign that this coming, see e.g. this 2019 DOE report, which declared scientific reproducibility in applications of scientific machine learning a priority research direction.

1
  • Thank you for your answer! I will check out the articles and reports you have mentioned
    – Max Muller
    Apr 20, 2020 at 19:37

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .