When Good and Reproducible Results are a Giant with Feet of Clay: The Importance of Software Quality in NLP (2024)

Abstract: Despite its crucial role in research experiments, code correctness is often presumed solely based on the perceived quality of results. This assumption, however, comes with the risk of erroneous outcomes and, in turn, potentially misleading findings. To mitigate this risk, we posit that the current focus on reproducibility should go hand in hand with the emphasis on software quality. We support our arguments with a case study in which we identify and fix three bugs in widely used implementations of the state-of-the-art Conformer architecture. Through experiments on speech recognition and translation in various languages, we demonstrate that the presence of bugs does not prevent the achievement of good and reproducible results, which however can lead to incorrect conclusions that potentially misguide future research. As countermeasures, we release pangoliNN, a library dedicated to testing neural models, and propose a Code-quality Checklist, with the goal of promoting coding best practices and improving software quality within the NLP community.

## introduction in the field of natural language processing (nlp), as well as in broader contexts, the validity and soundness of research findings are typically upheld by "establishing consistency of results versus existing implementations, standard benchmarks, or sanity checks via statistically significant experimental results" @xcite . embracing these recommendations as the exclusive criteria for validating scientific credibility, the research community has recently devoted significant attention to the reproducibility @xcite @xcite @xcite and the soundness of experimental settings and comparisons @xcite @xcite . specifically, in response to evidence indicating the absence of these aspects in many research papers @xcite and to mitigate the so-called "reproducibility crisis" @xcite ), foot_0 top-tier conferences have introduced dedicated checklists and targeted questions in the reviewing forms @xcite . however, a fundamental question remains regarding the initial assumption: are reproducibility and thorough evaluation against robust baselines sufficient to ensure the soundness of a research finding? according to peng (2011), reproducibility alone does not "guarantee the quality, correctness, or validity of the published results" since the code employed to produce them may not accurately execute its intended purpose. this entails inherent risks, as flawed code that produces good and easily reproducible results can propagate as the foundation for further research, ultimately leading to further unreliable and potentially misleading findings @xcite . expanding on these observations, this paper is a call to action, underpinned by empirical evidence, to bolster the dependability of nlp findings by complementing current initiatives toward reproducibility and experimental soundness with equal emphasis on software quality. to this end, we adopt as a reference framework the principles of software quality assurance @xcite , which have so far been overlooked by our community. building on this foundation, we contribute as follows: ## sqa and research software quality assurance (sqa) attributes have been studied for many years @xcite @xcite . delineated in the iso 9126 standard (iso/iec, 2001), they were later extended and superseded by iso 25010 (iso/iec, 2010), having production code as the main target. however, as they are desirable for any codebase, here we analyze how each attribute has an effect on research code and work. portability and usability refer, respectively, to the possibility of executing the same experiments in diverse hardware or software environments, and the effort required to use the software (i.e., how easy it is to run the code). as such, they pertain to the reproducibility of a paper, which, according to acm, foot_4 holds when "an independent group can obtain the same result using the author's own artifacts", a definition that is aligned with those given in nlp @xcite and other fields @xcite . regarding these aspects, ample literature already discussed the need to go beyond code openness in research @xcite , highlighting the role of proper documentation and validation in different environments. however, as many research groups lack access to a wide range of hardware options, we argue that research works can hardly target portability due to the significant economic and human resources it requires. on the contrary, proper documentation of the code is a reasonable demand and, in addition to increasing reproducibility, it facilitates code reuse and adoption for other works. code reusability also pertains to the maintainability attribute, which denotes the effort required for implementing targeted modifications. alongside comprehensive documentation, software maintainability hinges on code structure, i.e. the organization of the software into building blocks @xcite . while there is currently no incentive to develop reusable code @xcite , the research community would greatly benefit in the long term from a commitment to this objective, which would reduce the time spent in replicating prior work and accelerate the implementation of new techniques upon existing code. the expeditiousness of testing new methods also depends on the efficiency and reliability of the codebase. efficiency refers to the amount of resources a software uses, e.g. the number of gpu hours or vram gbs needed for training. increasing efficiency constitutes a research direction on its own and can hardly be considered a prerequisite for orthogonal investigations. reliability, instead, is the capability of the software to seamlessly operate in all conditions and for a long time: software causing frequent crashes (i.e. terminations due to errors) or whose efficiency is not constant over time is not reliable. although both properties would contribute to reducing the environmental footprint of nlp research by avoiding computing-resource wastes or unexpected failures @xcite , a commitment in this direction is arguably an excessive demand for research works not expressly dedicated to it. last but not least, functionality or functional correctness (hereinafter: correctness) pertains to the "extent to which a program satisfies its specifications" @xcite . in research, this holds when the code exactly performs the operations described in a paper, thereby establishing the validity of the reported findings. achieving correctness requires the creation and execution of tests, as they are the sole mechanism that guarantees the correct behavior of software. for example, when designing a causal model (i.e. a model that can-not look at future elements in the input sequence), researchers should test that the model predictions always remain unaffected by future elements. if a bug breaks the causality property, any observed gains may not stem from the proposed solutions but from undue access to forbidden information. it is worth emphasizing that the validity of these tests expires after any code alteration, regardless of its apparent relevance. therefore, tests should be executed after each modification to ensure correctness and, in turn, the trustworthiness of the findings. in summary, we have observed that, in the context of research software, i) portability and usability support reproducibility, ii) maintainability promotes reusability, iii) efficiency and reliability reduce environmental costs, and iv) correctness plays a crucial role in ensuring trustworthy findings and research soundness. however, despite its importance, we show in the next section that the research community has largely neglected correctness, focusing primarily on reproducibility. ## research code quality evaluation to assess the level of consideration given to the above sqa attributes within the nlp research community, we examined their inclusion in the review forms of top-tier conferences and journals in the field, namely: *acl (i.e., aacl-ijcnlp, acl, eacl, emnlp, naacl), foot_5 acl rolling review (arr), icassp, icml, iclr, interspeech, neurips, and tacl. we specifically focused on reproducibility (as a proxy of portability and usability) and correctness. most of the venues (5 out of 8) include an explicit score for reproducibility and neurips mentions it among the factors contributing to the overall recommendation score. reproducibility is commonly evaluated through dedicated checklists foot_6 that mainly focus on the detailed descriptions of the hyperparameters and the software/hardware environment (while disregarding whether different hardware/software is supported, i.e. portability, which seems reasonable as seen in §2). accordingly, these checklists are not strictly related to sqa, although they do include recommendations for proper code documentation, which is related to the software usability and maintainability. venue reproducibility correctness *acl arr icassp icml iclr interspeech neurips tacl correctness is instead mentioned in fewer forms (3 out of 8). when present, its definition varies and is not explicitly related to the code: at iclr and icassp, the scope of the term is not clearly defined, while in tacl it is included in the broader concept of soundness of the experiments/results. in this result-oriented definition, soundness pertains to assessing the significance of results (are the reported improvements robust to statistical fluctuations?) with respect to either the state of the art (are the results competitive with those reported in recent literature?) or strong baselines. soundness is also assessed at neurips and icml but, again, code correctness is never explicitly mentioned. notably, the interspeech form contains a "technical correctness" score, which however refers to the reproducibility of the paper ("are enough details provided to be able to reproduce the experiments?"). in general, when considered, software is explicitly evaluated only in terms of accessibility (is the code released open-source?) and potential usefulness (will the research community benefit from the use of the software?). for instance, the "software" score in the arr form only refers to the usefulness and documentation of newly-released code rather than to its correctness, thus being again more related to its usability and maintainability. we can conclude that, unlike reproducibilityrelated sqa attributes, correctness is largely neglected in favor of a result-based evaluation of soundness. from the researchers' perspective, this entails the risk of basing future work on unreliable software that yields high and easily reproducible results but lacks guarantees of its correctness. this risk, in turn, can lead to misleading findings @xcite . in the next section, we present a concrete instance of this problem with a case study analyzing open-source implementations of the widespread conformer architecture. ## the case study in our case study, we examine the conformer (gulati et al., 2020) architecture -the state-of-the-art solution for speech processing tasks @xcite @xcite such as automatic speech recognition (asr) and speech-to-text translation (st) -whose rapid and wide adoption is evidenced by more than 2,000 citations since 2020. 7 in the following, we first analyze the conformer implementation of six widely-used open-source codebases, showing that they all contain at least one bug ( §4.1). then, through extensive experiments on the two tasks and on eight language pairs, we demonstrate that the presence of bugs can be hidden by good -but incorrect -results ( §4.3), consequently leading to erroneous conclusions ( §4.4). an introduction of the asr and st tasks object of our study, along with an overview of the conformer architecture is provided in appendix a. ## increasing research code correctness after demonstrating that the current tendency to assess code correctness solely based on the reported results ( §3) potentially leads to wrong findings ( §4), in this section we propose countermeasures. specifically, we aim at fostering the adoption of sqa best practices in two ways: 1) by releasing a python package (pangolinn) for testing neural networks and assisting researchers in the verification of code correctness ( §5.1); 2) by proposing the integration of current conference checklists with recommendations for sqa best practices ( §5.2). ## conclusions in parallel with the current efforts to enhance the reproducibility of nlp research, this paper urges similar actions targeting the improvement of research software quality, underscoring its importance for the reliability of research findings. in comparison to the attention given to reproducibility within our community, we observed the predominant neglect of code correctness and elaborated on the risks associated with assessing soundness solely on the basis of experimental results. as we empirically demonstrated through a case study involving the widespread conformer architecture, such risks include the potential of drawing misleading conclusions from positive results obtained using flawed code. as a countermeasure, besides releasing a corrected conformer implementation, we created the pangolinn python package to facilitate testing neural models and proposed the adoption of a "code-quality checklist" aimed at fostering coding best practices. while we acknowledge that these solutions are not a panacea, their purpose is to raise awareness within the nlp community about the importance of software quality. we hope that our endeavor will inspire a collective commitment to developing high-quality and reliable code.