Two Reproductions of a Human-Assessed Comparative Evaluation of a Semantic Error Detection System (2022)

Abstract: In this paper, we present the results of two reproduction studies for the human evaluation originally reported by Dušek and Kasner (2020) in which the authors comparatively evaluated outputs produced by a semantic error detection system for data-to-text generation against reference outputs. In the first reproduction, the original evaluators repeat the evaluation, in a test of the repeatability of the original evaluation. In the second study, two new evaluators carry out the evaluation task, in a test of the reproducibility of the original evaluation under otherwise identical conditions. We describe our approach to reproduction, and present and analyse results, finding different degrees of reproducibility depending on result type, data and labelling task. Our resources are available and open-sourced.

## introduction reproduction studies are garnering growing interest in natural language processing (nlp), most recently as the subject of shared tasks @xcite . the importance of ensuring good levels of reproducibility in nlp work is increasingly recognised, and approaches to defining and assessing reproducibility are emerging @xcite . with this paper, we add to the growing body of reproduction studies by tackling a particularly hard case for reproducibility assessment, namely error analysis that involves identifying which of two disagreeing systems is making error(s), and further classifying the types of errors being made. we perform two reproductions, one involving the same evaluators as in the original study, one involving new evaluators. the former can be seen as a test of the repeatability of the original study, where nothing is changed except the point in time, and the latter as a test of its reproducibility where the reproduction differs from the original study in some specfied respect(s), here the evaluator cohort. 4 below we start by describing the original study and outlining our approach to reproduction. next we describe our two reproductions and present an analysis which examines three types of results from the evaluations, applying different tools for measuring similarity in each case. we finish with a discussion of the reasons behind and possible mitigation strategies for what is, on the face of it, a mostly poor set of reproducibility results. ## approach to reproduction for results of type i above (where we have single measured quantity values to compare), we follow the quantified reproducibility assessment (qra) approach @xcite which means (a) identifying and documenting (as we do in the attached heds sheet) the properties of evaluation experiments as standardised attribute-value pairs (conditions of measurement in qra terms); and (b) computing the small-sample coefficient of variation (cv * ) over compared quantity values, as the measure of degree of reproducibility. qra assessment results are shown in tables 2 and 3 and discussed in section 4.1. for type ii results ( for results of type iii, we compute fleiss's kappa (the multi-evaluator generalisation of scott's pi) on aligned sets of categorical values where we have exactly one label per item (which is the case for the correctness labels), and krippendorff's alpha where we have multiple labels per item (which is the case for the error labels). results are shown in ## two reproductions our two reproduction studies repeated the dušek and kasner evaluations as closely as possible, the first using the same evaluators, the second using different evaluators. there were two complicating factors, necessitating the use of (i) new evaluator instructions, and (ii) a different way of allocating evaluators to evaluation items. the reason for the difference in evaluator instructions is that in the original work, instructions were not written down, a shared understanding being evolved in the course of the work instead. in order to repeat the evaluations with new evaluators less familiar with the work, instructions had to be written down and shared which were then used in all reproductions. the instructions are included verbatim in the appendix. regarding evaluator allocation, in the original work, the work was shared between the two authors who each did about half of e2e and half of webnlg, but it was not recorded who did which ones. for that reason, we decided to get the evaluators in the reproduction studies (the original two authors, and authors 4 and 5 of this paper) to each annotate all 100 e2e items and all 100 webnlg items, and then we randomly selected half from each evaluator pair for a like-for-like comparison (in the tables below we call this the combined set of annotations). assessing the similarity between these combined results and the original results forms the main body of our reproduction study: type i results are shown in additionally, we compare the four complete sets of annotations with the original annotations individually, for the single numeric values (type i results) from e2e and webnlg only ( each evaluator worked on a separate google spreadsheet in the exact same format as in the original study, 8 except that in the repeatability test which involved the original annotators, we shuffled the order of evaluation items to avoid inadvertent recall of original annotations. ## discussion the error-analysis based evaluation method in this paper compares system outputs with reference outputs, but rather than just counting it against the system if there is disagreement between the two, it examines which is actually right in each case, also identifying the types of errors made by each. for e2e, 4 out of 5 sets of annotations ( other broad-strokes findings that are confirmed in all reproductions are that errors to do with pricerange predicate are the most common, and errors connected to eattype and famfriend are the least common, of the errors considered in e2e. for the error labels in webnlg no findings are supported by all sets of annotations. the degree to which the different types of results were reproducible varied. the more highlevel correctness labels saw far better agreement than the more fine-grained error labels which also involve greater cognitive load. moreover, the different backgrounds of the annotators and their degree of familiarity with the system and data may also have contributed to variation. it is likely that if our instructions had been more precise, and more training/discussions of annotators in interpreting the instructions had taken place, the variation between studies would have been lower, and we can see room for improvement in this respect which we plan to explore in future work, where we will aim to: the iterative annotation design and written guidelines would have been useful even for repeating the study with the original annotators, as even their annotations differed in the repeat. we also noted some ideas for improving the error classes, which probably would have been already implemented with an iterative approach. ## conclusion in this paper, we described two reproductions of a manual error analysis of the outputs from a semantic error detection (sed) system based on two-way entailment detection by an nli model. we selected three types of results for reproduction, namely single numeric values, sets of numeric values, and sets of discrete labels, each of which requires different methods of comparison. all three types of results have broadly similar degrees of reproducibility: higher-level findings are mostly confirmed but lower-level agreement measures show a more differentiated picture, and are particularly low for webnlg and error classes. results for e2e are generally better reproduced than webnlg, and correctness labels are easier to reproduce than the more fine-grained error classes. in terms of conclusions to be drawn from the reproduction studies reported here, as with many other reproductions we found that the details of design and execution of the original study had not been recorded at the level of detail required for a reproduction. as a field, nlp is not currently in the habit of recording design/execution details of human evaluations very comprehensively or testing reproducibility during method development, for which time and other resources are often cited as reasons. the latter would be mitigated by the use of standard methods and tools for recording details of experiments and for assessing reproducibility. an error related to the pricerange slot, e.g. incorrectly identifying a hallucination in the pricerange slot, when the price range information has in fact been correctly verbalised.