NLP Reproducibility For All: Understanding Experiences of Beginners (2023)

Abstract: As natural language processing (NLP) has recently seen an unprecedented level of excitement, and more people are eager to enter the field, it is unclear whether current research reproducibility efforts are sufficient for this group of beginners to apply the latest developments. To understand their needs, we conducted a study with 93 students in an introductory NLP course, where students reproduced the results of recent NLP papers. Surprisingly, we find that their programming skill and comprehension of research papers have a limited impact on their effort spent completing the exercise. Instead, we find accessibility efforts by research authors to be the key to success, including complete documentation, better coding practice, and easier access to data files. Going forward, we recommend that NLP researchers pay close attention to these simple aspects of open-sourcing their work, and use insights from beginners’ feedback to provide actionable ideas on how to better support them.

## introduction as natural language processing (nlp) research continues to grab public attention and excitement, it becomes increasingly important for it to be accessible to a broad audience. while the research community works to democratize nlp, it remains unclear whether beginners in the field can easily apply the latest developments. how easy is it for them to reproduce experimental results? will their programming background or comprehension of papers play a role? what key elements affect their experience here? to address these questions, we conducted a controlled user study in an introductory nlp class. we first identified and successfully reproduced the results of three recent nlp publications ourselves. we surveyed students on their background in machine learning and programming, then divided them into three skill level groups based on the survey results. each group was asked to reproduce the results of the three papers. as students conducted experiments, they tracked their own efforts, while their computational resource usage was logged by a shared high-performance computing system. after conducting the experiments, students answered questions and provided feedback about their experiences. our results show that beginners' technical skill level and comprehension of the papers play only a small part in their experience in reproducing the results, and we observed a strikingly wide range of time spent on the exercise regardless of these user-specific factors. meanwhile, we show that reproducibility efforts by paper authors make a much larger impact on user experience, and based on direct feedback from these beginners, we find that they encounter a range of roadblocks related to the documentation and ease of use of open-sourced materials. these findings shed light on the extra steps nlp researchers can take to make state-of-the-art technology more accessible to beginners, an important direction to continue democratizing nlp to its growing audience. to this end, we make several concrete recommendations in section 5 on how the nlp research community can further improve accessibility to beginners. ## related work amidst growing concern of a reproducibility crisis across many scientific disciplines (baker, 2016), recent years have seen an increasing amount of work studying trends and proposing guidelines to improve research reproducibility in nlp and neighboring disciplines (arabas et al., 2014; rozier and rozier, 2014; sethi and gil, 2016; henderson et al., 2018; crane, 2018; cohen et al., 2018; tatman et al., 2018; dodge et al., 2019; pineau, 2020; pineau et al., 2021; rogers et al., 2021; belz, 2022). as part of these efforts, the association for computational linguistics (acl) has also adopted author checklists for reproducibility and responsible research practice. as a prerequisite for reproducibility in nlp, prior work has studied the availability of code and data at scale (wieling et al., 2018). where code and data are available, large-scale and multi-test studies have assessed the reproducibility of results (antónio rodrigues et al., 2020; branco et al., 2020). various venues have invited individual reproductions and replications of work in machine learning and nlp, including the reproducibility, inexplicability, and generalizability of results workshop (arguello et al., 2016) in information retrieval, the reproducibility in ml workshops foot_0 at the international conference on machine learning (icml) and international conference on learning representations (iclr), and the ml reproducibility challenges foot_1 at iclr and neural information processing systems (neurips). belz et al. (2021) reviewed and aggregated many of these efforts in nlp, finding only up to 14% of experiments led to accurately reproduced results. unlike prior work, we are not focused on the accuracy of results, but rather nlp beginners' experiences in reproducing results that are pre-verified to be reproducible. ## methodology next, we introduce the methodology for our study, including the steps taken for data collection, and the variables of interest we study in our analysis of beginners' experience reproducing nlp results. ## results & findings here, we investigate how aspects of students' background and understanding of papers, as well as reproducibility efforts by the authors of papers, affected students' experience in reproducing experimental results. @xcite following prior work, we first verify the accuracy of the results. students on average produced results with a relative error within 3% accuracy from reported results regardless of their skill level or assigned paper. @xcite this is expected, as we verified that all papers' results can be accurately reproduced. next, we systematically investigate the experience students had reproducing results from the papers. we analyze their experience from three perspectives: students' skill level (section 3.2.2), students' comprehension of their assigned paper (section 3.2.3), and paper authors' efforts toward making their results more reproducible (section 3.2.4). ## discussion & recommendations our study reveals a wealth of insights into enhancing the accessibility of nlp research to beginners.