Quantified Reproducibility Assessment of NLP Results (2022)

Abstract: This paper describes and tests a method for carrying out quantified reproducibility assessment (QRA) that is based on concepts and definitions from metrology. QRA produces a single score estimating the degree of reproducibility of a given system and evaluation measure, on the basis of the scores from, and differences between, different reproductions. We test QRA on 18 different system and evaluation measure combinations (involving diverse NLP tasks and types of evaluation), for each of which we have the original results and one to seven reproduction results. The proposed QRA method produces degree-of-reproducibility scores that are comparable across multiple reproductions not only of the same, but also of different, original studies. We find that the proposed method facilitates insights into causes of variation between reproductions, and as a result, allows conclusions to be drawn about what aspects of system and/or evaluation design need to be changed in order to improve reproducibility.

## introduction reproduction studies are becoming more common in natural language processing (nlp), with the first shared tasks being organised, including re-prolang @xcite and reprogen @xcite . in nlp, reproduction studies generally address the following question: if we create and/or evaluate this system multiple times, will we obtain the same results? to answer this question for a given specific system, typically @xcite @xcite an original study is selected and repeated more or less closely, before comparing the results obtained in the original study with those obtained in the repeat, and deciding whether the two sets of results are similar enough to support the same conclusions. this framing, whether the same conclusions can be drawn, involves subjective judgments and different researchers can come to contradictory con-clusions: e.g. the four papers @xcite @xcite reproducing @xcite in reprolang all report similarly large differences, but only arhiliuc et al. conclude that reproduction was unsuccessful. there is no standard way of going about a reproduction study in nlp, and different reproduction studies of the same original set of results can differ substantially in terms of their similarity in system and/or evaluation design (as is the case with the vajjala and rama (2018) reproductions, see section 4 for details). other things being equal, a more similar reproduction can be expected to produce more similar results, and such (dis)similarities should be factored into reproduction analysis and conclusions, but nlp lacks a method for doing so. being able to assess reproducibility of results objectively and comparably is important not only to establish that results are valid, but to provide evidence about which methods have better/worse reproducibility and what may need to be changed to improve reproducibility. to do this, assessment has to be done in a way that is also comparable across reproduction studies of different original studies, e.g. to develop common expectations of how similar original and reproduction results should be for different types of system, task and evaluation. in this paper, we (i) describe a method for quantified reproducibility assessment (qra) directly derived from standard concepts and definitions from metrology which addresses the above issues, and (ii) test it on diverse sets of nlp results. following a review of related research (section 2), we present the method (section 3), tests and results (section 4), discuss method and results (section 5), and finish with some conclusions (section 6). ## related research the situation memorably caricatured by @xcite still happens all the time: you download some code you read about in a paper and liked the sound of, you run it on the data provided, only to find that the results are not the same as reported in the paper, in fact they are likely to be worse @xcite . when both data and code are provided, the number of potential causes of such differences is limited, and the nlp field has shared increasingly detailed information about system, dependencies and evaluation to chase down sources of differences. sharing code and data together with detailed information about them is now expected as standard, and checklists and datasheets have been proposed to standardise information sharing @xcite . reproducibility more generally is becoming more of a research focus. there have been several workshops and initiatives on reproducibility, including workshops at icml 2017 and 2018, the reproducibility challenge at iclr 2018 and 2019, and at neurips 2019 and 2020, the re-prolang @xcite initiative at lrec 2020, and the reprogen shared task on reproducibility in nlg @xcite . despite this growing body of research, no consensus has emerged about standards, terminology and definitions. particularly for the two most frequently used terms, reproducibility and replicability, multiple divergent definitions are in use, variously conditioned on same vs. different teams, methods, artifacts, code, and data. for example, for @xcite , reproducing a result means running the same code on the same data and obtaining the same result, while replicating the result is writing and running new code based on the information provided by the original publication. for @xcite , reproducibility is achieving the same results using the same data and methods. according to the acm's definitions (association for computing machinery, 2020), results have been reproduced if obtained in a different study by a different team using artifacts supplied in part by the original authors, and replicated if obtained in a different study by a different team using artifacts not supplied by the original authors. the acm originally had these definitions the other way around until asked by iso to bring them in line with the scientific standard (ibid.). conversely, in drummond's view 2009 obtaining the same result by re-running an experiment in the same way as the original is replicability, while reproducibility is obtaining it in a different way. @xcite , followed by schloss (2018), defines four concepts rather than two, basing definitions of reproducibility, replicability, robustness and generalisability on the different possible combinations of same vs. different data and code. none of these definitions adopt the general scientific concepts and definitions pertaining to reproducibility, codified in the international vocabulary of metrology, vim @xcite . one issue is that they all reduce the in principle open-ended number of dimensions of variation between measurements accounted for by vim to just two or three (code, data and/or team). another, that unlike vim, they don't produce comparable results. nlp does not currently have a shared approach to deciding reproducibility, and results from reproductions as currently reported are not comparable across studies and can, as mentioned in the introduction, lead to contradictory conclusions about an original study's reproducibility. there appears to be no work at all in nlp that aims to estimate degree of reproducibility which would allow crossstudy comparisons and conclusions. ## metrology-based reproducibility assessment metrology is a meta-science: its subject is the standardisation of measurements across all of science to ensure comparability. computer science has long borrowed terms, most notably reproducibility, from metrology, albeit not adopting the same definitions (as discussed in section 2 above). in this section, we describe quantified reproducibility assessment (qra), an approach that is directly derived from the concepts and definitions of metrology, adopting the latter exactly as they are, and yields assessments of the degree of similarity between numerical results and between the studies that produced them. we start below with the concepts and definitions that qra is based on, followed by an overview of the framework (section 3.2) and steps in applying it in practice (section 3.3). ## qra tests there are three sets of related systems: (i) the (single) pass football report generator (van der @xcite , (ii) vajjala and rama (2018)'s 11 multilingual essay scoring system variants, and (iii) two variants of @xcite 's neural text simplifier (nts). pass is evaluated with three evaluation measures (human-assessed clarity, fluency and stance identifiability), the essay scoring systems with one (weighted f1), and the nts systems with two (bleu and sari). for pass we have one reproduction study, for the essay scorers seven, and for the nts systems, from three to six. the pass reproduction was carried out as part of reprogen @xcite , the reproductions of the essay-scoring systems and of one of the nts systems as part of reprolang @xcite , and we carried out an additional reproduction study of the nts systems for this paper. 8 the pass text generation system is rule-based, the essay classifiers are 'theory-guided and datadriven' hybrids, and the text simplifiers are end-toend neural systems. this gives us a good breadth ## discussion quantified reproducibility assessment (qra) enables assessment of the degree of reproducibility of evaluation results for any given system and evaluation measure in a way that is scale-invariant foot_8 and comparable across different qras, for reproductions involving either the same or different original studies. moreover, formally capturing (dis)similarities between systems and evaluation designs enables reproducibility to be assessed relative to such (dis)similarities. in combination, a set of results from qra tests for the same system and evaluation measure can provide pointers to which aspects of the system and evaluation might be associated with low reproducibility. e.g. for the wf1 evaluations of the essay scoring systems above, it is clear that variations in reproducibility are associated at least in part with the different features used by systems. it might be expected that the reproducibility of human-assessed evaluations is generally worse than metric-assessed. our study revealed a more mixed picture. as expected, the fluency and clarity evaluations of the pass system were among those with highest cv * , and the bleu and sari evaluation of the nts systems and wf1 evaluation of the mult-pos and mult-dep systems were among those with lowest cv * . however, human-assessed stance identifiability of pass was among the most reproducible, and metric-assessed wf1 of mult-base, mult-dom and mult-emb were among the worst. in this paper, our focus has been qra testing of existing research results. however, ideally, qra would be built into new method development from the outset, where at first reporting, a detailed stan- if this is not possible, post-hoc reproducibility qra (where there are differences in conditions of measurement values) is performed instead. if this yields high (poor) cv * , one way to proceed is to minimise differences in conditions of measurement between the studies and observe the effect on cv * , changing aspects of system and evaluation design and adding further conditions of measurement if need be. for human evaluation in particular, persistently high cv * would indicate a problem with the method itself. ## conclusion we have described an approach to quantified reproducibility assessment (qra) based on concepts and definitions from metrology, and tested it on 18 system and evaluation measure combinations involving diverse nlp tasks and types of evaluation. qra produces a single score that quantifies the degree of reproducibility of a given system and evaluation measure, on the basis of the scores from, and differences between, multiple reproductions of the same original study. we found that the approach facilitates insights into sources of variation between reproductions, produces results that are comparable across different reproducibility assessments, and provides pointers about what needs to be changed in system and/or evaluation design to improve reproducibility. a recent survey @xcite found that just 14% of the 513 original/reproduction score pairs analysed were exactly the same. judging the remainder simply 'not reproduced' is of limited usefulness, as some are much closer to being the same than others. at the same time, assessments of whether the same conclusions can be drawn on the basis of different scores involve subjective judgments and are prone to disagreement among assessors. quantifying the closeness of results as in qra, and, over time, establishing expected levels of closeness, seems a better way forward.