Generation Challenges: Results of the Accuracy Evaluation Shared Task (2021)

Abstract: The Shared Task on Evaluating Accuracy focused on techniques (both manual and automatic) for evaluating the factual accuracy of texts produced by neural NLG systems, in a sports-reporting domain. Four teams submitted evaluation techniques for this task, using very different approaches and techniques. The best-performing submissions did encouragingly well at this difficult task. However, all automatic submissions struggled to detect factual errors which are semantically or pragmatically complex (for example, based on incorrect computation or inference).

## introduction users expect data-to-text natural language generation (nlg) systems to generate textual summaries which are accurate. however, many nlg systems, especially neural ones, generate texts which are factually incorrect. the most reliable way to assess the accuracy of a generated text is to ask human annotators to carefully fact-check the text. however this is a time-consuming and expensive process. in earlier work, we developed a protocol @xcite where three mechanical turk workers (who had been screened and passed a qualifying test) carefully annotated factual errors in a text produced by a neural nlg system. the protocol was effective and showed high interannotator agreement, but it took annotators 20-30 minutes (each) to fact-check a moderately complex 300-word paragraph produced by a neural data-to-text nlg system. the total cost of the process (including fees to amazon and money spent on the screening process for potential annotators) was about us$30 per text. it would be very useful to the nlg community if we could come up with quicker and easier ways of measuring accuracy and factual correctness which have good correlations with the protocol of @xcite . such methods could be based on less time-consuming human evaluations or on automatic metrics. however, these techniques should only be used if they have good agreement and correlation with careful high-quality human fact-checking by multiple annotators. in this shared task, participating teams submitted techniques (both human and automatic) for evaluating the factual accuracy of summaries of basketball games produced from box score (and other game data) by three neural nlg systems. these techniques were evaluated by computing precision and recall (of identified factual errors) against a goldstandard human annotation produced by @xcite 's protocol. some of the systems did well overall, but it was also clear that some types of factual errors are difficult to detect. we hope that our shared task encourages researchers from many fields to work on the problem of identifying factual errors in generated texts; progress in this area would be very helpful for nlg. full details of the shared task requirements, as well as both the training and test corpus can be found at https://github.com/ehudreiter/accuracysharedtask . ## task participants were asked to submit a technique for identifying incorrect statements in a generated text. this meant statements which are not true in the real world (ie, classic fact-checking), not just statements which disagree with (or are not derivable from) the system run-time data (see section 3.1 of @xcite ). techniques could be • human evaluation protocols. subjects would have access to data about the game and the teams, and also (if part of the protocol) to a human-authored reference text. the output of the evaluation protocol or metric was a list of mistakes in the text. each mistake was characterised by thomson and reiter (2020) -incorrect number: it does not matter whether the number is spelled out or is in digits. -incorrect name (for named entities): in a sports reporting context, this includes people, places, teams, and days of the week. -incorrect word: a word which is not one of the above and is incorrect. -context error: a phrase which causes an incorrect inference because of context or discourse. -not checkable: a statement which can not be checked, either because the information is not available or because it is too time-consuming to check. -other: any other type of mistake. an example is shown in figure 1 . note that this example combines fragments from texts produced by several different systems, along with some manual adjustments, in order to illustrate different types of mistakes in a simple way. ## data we manually annotated, using the procedure of @xcite , 90 texts produced by three neural nlg systems that use basketball box score data: @xcite , @xcite , and @xcite . in total, 30 texts were annotated from each system. of these, 60 texts (20 from each system) were given to shared task participants as training data, and 30 texts (10 from each system) were reserved for a separate test set, which participants did not see until they had submitted their solutions. the memphis grizzlies (5-2) defeated the phoenix suns (3 -2) monday 102-91 at the talking stick resort arena in phoenix. the grizzlies had a strong first half where they out-scored the suns 59-42. marc gasol scored 18 points, leading the grizzlies. isaiah thomas added 15 points, he is averaging 19 points on the season so far. list of errors: ## accuracy errors observed in this section we discuss and give some insights about the accuracy errors we observed in the manually-annotated training data (i.e, the 60 annotated texts given to participants as training data). we look separately at the different types of errors listed in section 2, and also at the impact of position and the neural nlg system used. ## results the submissions were evaluated by computing their recall and precision against the gold-standard mistake list (gsml) which was based on the human annotated texts in the test set (section 3). in other words, for each submission, we calculated how many of the gold-standard mistakes were detected by that submission (recall), and how many of the mistakes detected by that submission were present in the gold-standard annotation (precision). we calculated this at the level of both mistakes and tokens. tables 4 to 8 show recall/precision of the submissions for different types of mistakes, as well as overall. we can see that the automatic techniques (tables 5 to 8 ) were unable to detect context and other errors, and only the laval university (metric) system could detect not checkable errors (but at low precision and recall). we can also see that none of the automatic systems did well at detecting word errors; the best system, charles-upf, had around 50% precision and recall. overall, this suggests that semantically more complex errors are harder to detect automatically, which is not surprising. as a point of comparison, the relation generation metric @xcite , which has been widely used by many previous papers to evaluate accuracy, can only detect name and number errors and has a recall of less than 40% for these types of errors @xcite . this is considerably worse than the best-performing submissions to our shared task. player groups: 6 mistakes incorrectly described a group of players, such as a duo. 45% of blind spot mistakes involved word, context, and not-checkable errors, compared to only 30% overall in the gsml. in addition, only 8% of blind spot mistakes were cardinal numbers, despite these constituting 33% of the gsml. it is important that we do not miss blind spot mistakes as whilst they are only 14% of the current gsml, this proportion could increase as systems become better at avoiding simple errors. ## conclusion neural data-to-text systems need to be able to produce accurate texts in order to be genuinely useful in most nlg use cases. an essential prerequisite to improving accuracy is being able to measure and evaluate accuracy. we believe that the evaluation techniques submitted to our shared task represent a major advance in the state of the art. we encourage participants and others to continue developing better-performing techniques for this key evaluation task.