ACL Tutorial Proposal: Towards Reproducible Machine Learning Research in Natural Language Processing (2022)

Abstract: While recent progress in the field of ML has been significant, the reproducibility of these cutting-edge results is often lacking, with many submissions lacking the necessary information in order to ensure subsequent reproducibility. Despite proposals such as the Reproducibility Checklist and reproducibility criteria at several major conferences, the reflex for carrying out research with reproducibility in mind is lacking in the broader ML community. We propose this tutorial as a gentle introduction to ensuring reproducible research in ML, with a specific emphasis on computational linguistics and NLP. We also provide a framework for using reproducibility as a teaching tool in university-level computer science programs.

## breadth of the tutorial in the tutorial, we introduce and contrast reproducibility @xcite , discuss papers reflecting on the reproducibility crisis in ml and nlp @xcite @xcite , including possible reasons for this crisis @xcite . this includes barriers to reproducibility, such as lack of code availability @xcite and the influence of different experimental setups @xcite @xcite . @xcite investigates the reproducibility of ml papers without accessing provided code, relying on only details provided in the paper. @xcite attempt to quantify reproducibility in nlp and ml. we also discuss reproducibility checklists from multiple venues @xcite @xcite @xcite . finally, we discuss coursework focused on teaching through repro-1 https://paperswithcode.com/rc2021 ducibility in ml @xcite and fact-ai @xcite . ## reading list we briefly describe recommended reading for participants in this section. ## ethics statement reproducibility and ethics are inherently related, since ensuring that research is reproducible by members of the community that are not its original authors contributes to making the field more inclusive (e.g. providing the code and hyperparameters needed to replicated a state-of-the-art ml model can help researchers build and expand upon it). furthermore, being transparent about the costs of the model, both in terms of the computational power need to train it as well as the data involved, helps members of the community be more equitable in evaluating it: for instance, if two models achieved similar accuracy on the same dataset, with one requiring 10x more computation than the other, that could help researchers choose which one to use given their constraints. finally, progress in the field of computational linguistics specifically is being led by large organizations that are the ones training and deploying equally large language models that are difficult to replicate without having access to the same resources that they do; being more transparent and ensuring that even large language models are replicable is important for making the field more democratic as a whole. ## pedagogical material as mentioned in section 3.4, we want instructors to be able to use content from our tutorial in order to design reproducibility projects for graduate-level coursework. the content will largely be based on the following components: (i) a blog post on how to use the ml reproducibility challenge as an educational tool @xcite , (ii) blog post on one university's experience in using the ml reproducibility challenge as an educational tool @xcite , and (iii) the corresponding paper @xcite . we hope this can function as a starter pack for any instructor who is interesting in incorporating reproducibility projects in their coursework. ## presenter information ana lucic is a phd candidate at the university of amsterdam. her work primarily focuses on developing and evaluating methods for explainable machine learning (ml). she co-developed a graduate-level course called fairness, accountability, confidentiality and transparency in artificial intelligence (fact-ai) that is centered around reproducing existing fact-ai algorithms. her email is a.lucic@uva.nl. maurits bleeker is phd candidate at the university of amsterdam who co-developed the fact-ai course. his main interest lies in the development of new optimization functions for image-text matching, by taking task-and data-specific inductive priors into account. this with the goal to improve the computational efficiency of multi-modal optimization. he also co-developed and coordinated two iterations of the fact-ai course at the university of amsterdam. his email is m.j.r.bleeker@uva.nl. samarth bhargav is a phd candidate at the university of amsterdam. samarth's research focuses on representation learning for information retrieval, with a goal of making ir systems (e.g recommenders) more amenable to user control, for example, through conversational interfaces. his secondary interests include recommendation in a cross-market or cross-domain setting, known-item retrieval, fact in ir and teaching ir. he has co-developed and taught multiple iterations of graduate ir courses at the university of amsterdam. his email is s.bhargav@uva.nl. jessica zosa forde is a phd candidate at brown university. jessica's research focuses on the empirical study of deep learning models, to improve their reliability in high stakes domains such as healthcare. she has also studied the inductive bias of overparameterized models, and model pruning. she believes that the open science movement is important for improving transparency and accountability in ml. she is also am a co-organizer of the ml reproducibility challenge (mlrc) and the ml retrospectives workshop. her email is jessica_forde@brown.edu. koustuv sinha is a phd candidate at mcgill university/mila. he is the lead organizer of the annual ml reproducibility challenge (mlrc), which has had five iterations since 2018 (at iclr 2018, iclr 2019, neurips 2019, mlrc 2020, mlrc 2021). he also serves as an associate editor of rescience, a journal promoting reproducibility reports in various fields of science. koustuv's research focuses on investigating systematicity in natural language understanding (nlu) models, especially the state-of-the-art large language models. his research goal is to develop methods to analyze the failure cases in robustness and systematicity of these nlu models, and develop methods to alleviate them in production. his email is koustuv.sinha@mail.mcgill.ca. sasha luccioni is a research scientist at huggingface. she has been an organizer of the ml reproducibility challenge (mlrc) since 2021 and is an area chair for the ethics in nlp track at emnlp 2021. sasha's research aims to contribute towards understanding the data and techniques used for developing machine learning approaches. she is particularly interested in developing tools for analyzing and filtering the data used for training large language models, as well as quantifying their carbon footprint. she has lectured several classes in ml and nlp, and is the main instructor for the forthcoming deeplearning ai "ai for social good" course. her email is sasha.luccioni@huggingface.co.