The Accuracy Evaluation Shared Task as a Retrospective Reproduction Study (2022)

Abstract: We investigate the data collected for the Accuracy Evaluation Shared Task as a retrospective reproduction study. The shared task was based upon errors found by human annotation of computer generated summaries of basketball games. Annotation was performed in three separate stages, with texts taken from the same three systems and checked for errors by the same three annotators. We show that the mean count of errors was consistent at the highest level for each experiment, with increased variance when looking at per-system and/or per-error- type breakdowns.

## introduction to address issues of factual accuracy in data-to-text systems, we developed a protocol for annotating mistakes in nlg texts @xcite . this protocol was used to create a corpus of errors found in generated basketball game summaries. the corpus was then used in the accuracy evaluation shared task @xcite , where participants submitted automatic metrics or alternative human evaluations that were compared to the gold standard. the corpus was created in three stages under experimental conditions which were largely the same. the same user interface and platform (amazon mechanical turk) were used, along with the same three annotators who each checked every game summary in all experiments. the only changed conditions were the game summaries that were checked, and slight clarification of the instructions based on annotator queries and feedback. therefore, it can be retrospectively considered a reproduction attempt. the original goal of these human evaluations was to design and develop a reliable protocol, then create gold list of errors (for training metrics, etc). in @xcite we performed an initial run of the protocol, with two subsequent runs in @xcite to extend the first run to form a training set, and then create a test set in the third run for use in a shared task. generated texts were annotated in equal proportions, within each experiment, from three different systems @xcite @xcite which at the time were representative state of the art systems on the rotowire dataset @xcite foot_0 of english language basketball summaries paired with box score data tables. in this paper we examine whether a similar number of errors were obtained in each experiment. we do this for all systems combined, the ensemble of representative errors that we intended to collect, and also at the per-system level, where we look at the errors for each system in isolation. we also discuss the issues that might be encountered when trying to reproduce or otherwise verify results obtained using the gold standard protocol for factual accuracy. ## related work it is crucial that our evaluation protocols are reliable, something that can be demonstrated by reproducing experimental results under similar conditions. such reproduction work is seldom carried out within the field of nlp @xcite . when it is, researchers experience difficulties obtaining the same results or finding the information required to run the experiment at all @xcite . problems with reproduction are not limited to nlp. in a large scale survey of over 1,500 researchers, @xcite found that 90% felt there was a reproducibility crisis, with over 50% deeming it 'significant'. the reprogen shared task, for which this paper is a submission, aims to document reproduction attempts in nlp and provide an improvement in levels of reproducibility over time. ## the gold standard protocol for factual accuracy the gold standard protocol detailed in @xcite uses human annotators to check the factual accuracy of generated texts. as part of this work, basketball games summaries were annotated for factual errors. these summaries are based on complex data, often including information from outwith the game being summarised, such as aggregated statistics or upcoming game schedules. this presents problems of error detection that are not found in simpler tasks. fact checking is performed against a comprehensive external data source rather than system input data, i.e., annotators check whether the text truthfully reflects what actually happened in the basketball games. full details can be found in @xcite and @xcite , although a brief overview is included here. ## experiment setup generated basketball summaries from the same three systems were used in each experiment. the systems were the conditional copy system of wiseman et al. (2017) , the document plan system of @xcite , and the hierarchical encoder of @xcite . these systems were chosen because each was considered stateof-the-art (by one or more metrics) at the time of publication. generated game summaries were provided by the authors of each paper, with the original rotowire dataset and partitions having been used. the set of distinct games from the rotowire test set was taken then randomly converted to a list. selection of games from within this random list was arbitrary, with games for the training gsml taken from the start of the list, and those for the test gsml taken working backwards from the end. each input game record was processed by only one system, therefore there was no comparison between systems of generated texts for the same game data. this was because the original goal was the development of annotation techniques and a list of gold errors, and not the comparison of different systems. when comparing systems retrospectively as we are in this paper, we do so with this caveat. the experiments we performed to collect data for the shared task were were: experiment c: 30 texts, 10 per system (test set). collected in march 2021. ## results we calculated the mean error counts (total errors by documents in experiment), as well as the coefficient of variation, cv* foot_4 @xcite . see our repository foot_5 for complete code and data, including the calculation of mean errors from the gsml. all values are calculated then rounded to two decimal places for inclusion in tables here. 1 than the mean error count (mec) had low variance between experiments, with a coefficient of variation of 3.61. this is what the experiments had originally set out to do; acquire representative samples of errors from neural systems. that similar quantities were found from the same ensemble of systems within each experiment is reassuring. ensemble per-type errors: when looking at the per-type breakdown for ensemble errors ( system per-type errors: when looking at the per-type breakdown for per-system errors, we see in figure 2 shows the spread of per-document error means for each system, within each experiment. it is worth noting that no generated text was error free and they rarely had fewer than 10 errors. ## discussion the experiments showed that when taking an ensemble of 3 models to create the gsml, the mean error count remained relatively stable between experiments. this adds to the evidence of the gold standard protocol being a reliable method of obtaining instances of errors which can then be used to evaluate alternative methods, such as metrics @xcite @xcite or cheaper human evaluations @xcite . the level of reproducibility for the gold standard protocol when evaluating systems is harder to determine. with a small number of texts per system in each experiment, the means are susceptible to the effects of outlier documents, such as rare cases where the document had 50 or more errors. the per-system coefficient of variation of ranged from about 10 to 13, which is similar to some cv* values reported for other human evaluations @xcite . the per-type results are limited by the low frequency of some types, but also by the complex resolution problem. in some cases there can be many correct ways in which a text can be annotated for errors, using different combinations of error types. an alternative way to measure the reproducibility of the protocol would be to run all three experiments again with different annotators. we could then look at how the sets of errors from the original experiment and the reproduction overlap. however, the problem of complex error resolution rears its head again. just because annotation spans or categories differ, does not necessarily mean that both sets of annotators did not correctly identify the same underlying problem. this addresses the issues of complex error resolution in a way which the exact comparison of token spans and labels does not. error verifiers can be asked to consider whether a reported error is one valid way to indicate the underlying problem. for discussion of complex annotation see @xcite . measuring reproducibility allows us to determine whether our evaluation protocols are reliable. however, it is not the only method for doing so. an alternative for validating the gsml would be to show individual errors to participants that are familiar with the annotation process, then ask them to indicate whether the highlight represents an error. this would allow us to measure the precision of annotators. we might also check in the same way, any errors reported by a minority of annotators. this would determine whether these errors were false positives, simply missed by the other annotators, or the result of differing annotations for complex errors. ## conclusion this reproduction study showed that there was little variance in the mean error count between the different experiments that were used for the shared task data collection. increased variance was observed when comparing the mean counts of different error types, and/or when comparing systems. these values do not, however, tell the whole story of this detailed evaluation protocol. for annotation based approaches the agreement between annotators can be measured @xcite , although with complex data-to-text, a lack of measurable agreement (based on token overlap) does not necessarily mean that annotators did not find similar underlying problems. an alternative when working at the level of individual errors might be to verify each reported error by asking additional annotators whether they agree with the reported error.