Shared Task on Evaluating Accuracy (2020)

Abstract: We propose a shared task on methodologies and algorithms for evaluating the accuracy of generated texts, specifically summaries of basketball games produced from basketball box score and other game data. We welcome submissions based on protocols for human evaluation, automatic metrics, as well as combinations of human evaluations and metrics.

## introduction users expect data-to-text nlg systems to generate textual summaries which are accurate. however, many neural nlg systems in particular generate texts which are factually incorrect. the most reliable way to assess the accuracy of a generated text is to ask human annotators to carefully fact-check the text. however this is a time-consuming process. our experiences at aberdeen @xcite show that it can take an experienced annotator 30 minutes to factcheck a moderately complex 300-word paragraph produced by a neural data-to-text nlg system. it would be very useful to the nlg community if we could come up with quicker and easier ways of measuring accuracy which have good correlations with careful fact-checking. such methods could be based on less time-consuming human evaluations, such as asking subjects to rate the accuracy of a text on a likert-type scale @xcite , or on automatic metrics. however, we should only use such techniques if we feel confident that they have good agreement and correlation with careful high-quality human fact-checking. the goal of our proposed shared task is to encourage innovative ideas for evaluating accuracy, including both automatic metrics and protocols for human evaluation. participants will apply their techniques to summaries of basketball games produced from box score (and other game data) by neural nlg systems. from the output of three such systems we will compile a corpus of generated texts. the shared task is unusual because submissions can be protocols for human evaluations as well as computer algorithms (ie, metrics). the community has limited experience with shared tasks which evaluate protocols, and we hope our experiences will help develop a better understanding of how to undertake such shared tasks, as well as a better understanding of how to evaluate the accuracy of nlg texts. ## organisers it is acceptable for submissions to give human subjects or metrics access to additional data beyond the box score and other game data used to generate the texts at run-time. the goal of the shared task is to find statements which are not true in the real world (ie, classic fact-checking), not just statements which disagree with (or are not derivable from) the system run-time data (see section 3.1 of @xcite ). the output of the evaluation protocol or metric will be a list of mistakes in the text. each mistake will be characterised by thomson and reiter (2020) -incorrect number: it does not matter whether the number is spelled out or is in digits. -incorrect named entity: this includes people, places, teams, and days of the week. -incorrect word: a word which is not one of the above and is incorrect. -context error: a phrase which causes an incorrect inference because of context or discourse. -not checkable: a statement which can not be checked, either because the information is not available or because it is too time-consuming to check. -other: any other type of mistake. an example is shown in figure 1. note that this example combines fragments from texts produced by several different systems, along with some manual adjustments, in order to illustrate different types of mistakes in a simple way. box score and other data for this game is available at https://www.basketball-reference. com/boxscores/201411050pho.html . we will also ask participants to submit estimates of the time required to find mistakes in a text (human time for human evaluations, and cpu/gpu time for metrics). this is optional, it is not required. it would, however, be very useful information for the nlg community, the memphis grizzlies (5-2) defeated the phoenix suns (3 -2) monday 102-91 at the talking stick resort arena in phoenix. the grizzlies had a strong first half where they out-scored the suns 59-42. marc gasol scored 18 points, leading the grizzlies. isaiah thomas added 15 points, he is averaging 19 points on the season so far. list of errors: ## data we will use texts produced by three systems that use basketball box score data: wiseman et al. (2017), puduppully et al. (2019a), and rebuffel et al. (2020 ). we will carefully fact-check, using the protocol of @xcite , 60 texts (twenty from each system). the three systems we have chosen all explored different ways of modifying the neural architecture. the system of @xcite defined the rotowire task and provided initial benchmarks for machine translation systems using copy attention, it is included for this reason. @xcite jointly conditioned on a document plan, whilst @xcite used a hierarchical encoder to group attributes (such as statistics) by their respective entities (players/teams). other systems in this domain which could be used for evaluation include @xcite , @xcite , @xcite . our aim, however, is to assess how well results produced by the participant's evaluation techniques correlate with the gold-standard fact-checking. hence we are looking for a set of systems which generate texts that contain a significant number of accuracy errors, not complete coverage of all systems that generate texts from basketball box score data. in @xcite , we looked at a number of texts generated by the three systems we have selected. no text was error free (the lowest was 7 errors) and the average number of errors per text is about 20. we will also ask each participant in the shared task to manually fact-check an additional ten texts. this is optional, but we believe it is very useful for building a better understanding of the task, as well as increasing the amount of training data available. note that the protocol of @xcite asks for each text to be fact-checked by three annotators. since this is expensive, and @xcite report a high-level of inter-annotater agreement, we will accept contributions from participants which have been factchecked by just one person. in total, we hope to have 100 fact-checked texts in the training set. participants will also have access to all of the texts produced by each of the three systems, along with source box score data and a human-written reference text. we will create a separate test set of 21 texts which will be manually fact-checked. the test set will include 7 texts from each of the above three systems. each text will be fact checked by 3 annotators following the same process as for the training data. ## evaluation plans we will release the test set (but not the manual factchecking annotations), and give participants two weeks to apply their techniques to the test set and return the results. each mistake will be reported as a position and category, as described above. we will create a reported mistake list (rml) for each annotated text submitted by a participant. we will then try to align each rml entry with an entry in the gold standard mistake list (gsml). match criteria will be applied in order, to each reported mistake (rm) in the rml. we will mark each match, along with which criteria it was identified under. in the event of an rm not being found in the gsml (i.e. a false positive), we will mark 'not found'. the match criteria are as follows: ## schedule we plan on the following schedule. note that dates are tentative and may be modified when we know the dates of inlg 2021.