Reproducibility of Exploring Neural Text Simplification Models: A Review (2022)

Abstract: The reproducibility of NLP research has drawn increased attention over the last few years. Several tools, guidelines, and metrics have been introduced to address concerns in regard to this problem; however, much work still remains to ensure widespread adoption of effective reproducibility standards. In this work, we review the reproducibility of Exploring Neural Text Simplification Models by Nisioi et al. (2017), evaluating it from three main aspects: data, software artifacts, and automatic evaluations. We discuss the challenges and issues we faced during this process. Furthermore, we explore the adequacy of current reproducibility standards. Our code, trained models, and a docker container of the environment used for training and evaluation are made publicly available.

## introduction in a survey conducted among 1,576 scientific researchers by nature @xcite , 90% believed that there is at least a slight crisis when it comes to the reproducibility of research. although there are no concrete statistics, the quantity and the growth of machine learning publications that rely on empirical evidence have recently raised alarms. to improve reproducibility, this community has designed checklists @xcite @xcite , guidelines (acm, 2022), and challenges @xcite , which highlight the importance of reproducibility, encourage best practices, and create a platform for conducting reproducibility studies. still, measures of reproducibility "in the wild" (that is, pertaining to real, widely cited machine learning and natural language processing studies) are limited. in this work, we set out to reproduce one such study as a case example to provide a concrete measure of reproducibility for a specific work. we select exploring neural text simplification models by @xcite . this paper poses an intriguing case: the research artifacts released by the authors are of high quality, the details they have provided match or exceed the current reproducibility recommendations, and two other reproducibility studies @xcite have successfully reproduced the results with high precision. reviewing a highquality scientific publication that has been the focus of multiple reproducibility studies enables us to build and expand upon those works. our primary objective is to investigate the ease with and extent to which the selected paper can be reproduced. we limit introducing new configurations, adding them only to cases necessary for further understanding the reproducibility results and not for competing scenarios. in section 2, we present the background, the task itself, the model, and its variants. in section 3, we describe our methodology and the steps we take to review the reproducibility of @xcite 's work. in particular, we look at associated data, software artifacts, and automatic evaluations. we present our results in section 4, before concluding by discussing our findings and recommendations for addressing the shortcomings of current checklists (section 5). we release our reproducibility artifacts to facilitate and promote future reproducibility studies @xcite . these artifacts include the updated source code, trained model, and complete runtime environment in a self-contained docker container. ## methodology there is no standard protocol or set of guidelines for conducting a reproducibility study, so we rely on the best practices suggested by others to provide a subjective and objective evaluation of the reproducibility of @xcite 's work. these best practices originate from checklists @xcite , research focused on reporting and evaluation @xcite @xcite , and reproducibility tracks at conferences and workshops @xcite . although we do not fill out any checklists, as they are not created for the purpose of third-party evalu- split sentences train (ew-sew) 284,677 validation (turkcorpus) 2,000 test (turkcorpus) 359 in the following subsections, we first examine the data used for this work and then shift our attention to the released software artifacts. we perform these preliminary steps to identify potential obstacles to reproducibility and to test the adequacy of the existing standards. later, we assess the reproducibility of the reported automatic evaluations. ## results in this section, we describe the outcomes of our reproducibility study in terms of data, software artifacts, and automatic evaluation. ## discussion taking all our experiments into account, we cannot claim that the performance difference between different variants comes from the design decisions made during their development have observed that changes to random seeds can result in substantially different results. perhaps our most surprising finding is that the nts-w2v variants affected by mismatched embeddings performed on par with the other variants once training was complete. this extreme level of resilience is, in fact, quite alarming. nearly all publications utilizing neural networks report top-performing empirical results; yet, aside from manual code review and deep analysis of the final results, there are no other clear signs or warnings that may suggest a bug is impacting the model. in this case, we found that our findings from the random seed experiments and validation performance during the training process were the only indicators that something was amiss. we recommend that future studies include random seed analysis demonstrating the range of the results that can be achieved with varied seeds, although we recognize that this search is the most expensive in terms of computation and may not be feasible in every case. perhaps due to the age of this repository, getting the project to a running state consumed the most time. we suspect that the situation will deteriorate as most dependencies are no longer being actively maintained. researchers should be hesitant with introducing new dependencies into their projects. additionally, we believe it would be fruitful to redirect the time and effort used for identifying and reporting dependencies toward exporting self-contained environments. this is an inadequacy that we found in nearly all of the checklists; in the case of this project, even though we knew all the requirements, we spent hours debugging different errors. the reproducibility of a reproducibility study is equally important, if not more than the reported findings. while the contribution of the original paper includes a novel idea, our goal was to provide a final artifact having the highest possible degree of reproducibility, and to assess the ease with and extent to which the selected paper could be reproduced. it would be interesting to return and perform a meta-analysis of this work in a few years to see how much the claims hold over time. while it is impossible to stop hardware and software from changing constantly, there are steps that can be taken in order to prolong the lifespan of a research artifact. we have made changes and fixes publicly available in a forked repository of the original paper. 7 additionally, we exported and released a self-contained docker container capable of training and running the model without any internet access @xcite . lastly, all the trained models are available for download. 8 despite all these attempts, it is hard to predict future problems that might occur. even the docker container depends upon the host environment (particularly, the kernel, gpu driver, and the docker itself). we have released our full runtime environment through zendo @xcite . ## conclusions in this paper, we reviewed the reproducibility of exploring neural text simplification models by @xcite . in our three step process, we analyzed the reproducibility of the data, the software artifacts, and the automatic evaluations. we would have liked to analyze the reproducibility of human evaluations given additional time. we hope that our released artifacts offer other researchers a head start for future reproducibility studies.