A Gold Standard Methodology for Evaluating Accuracy in Data-To-Text Systems (2020)

Abstract: Most Natural Language Generation systems need to produce accurate texts. We propose a methodology for high-quality human evaluation of the accuracy of generated texts, which is intended to serve as a gold-standard for accuracy evaluations of data-to-text systems. We use our methodology to evaluate the accuracy of computer generated basketball summaries. We then show how our gold standard evaluation can be used to validate automated metrics.

## introduction in most contexts, it is essential that texts produced by data-to-text natural language generation (nlg) systems accurately communicate input data. hallucination and other forms of inaccuracy are unacceptable in nlg application contexts such as journalism, financial reporting, and medical patient information. for example, it is not acceptable to give a doctor incorrect information about a patient. this means that it is essential that nlg developers be able to evaluate whether texts produced by their systems are accurate or not. we propose here a methodology (protocol) for high-quality human evaluation of the accuracy of generated texts. the methodology focuses on identifying and categorising specific accuracy errors in a text; hence it is quite different from protocols which ask people to assess the overall accuracy of a text on a scale existing work on detecting mistakes and hallucinations in nlg texts has largely focused on short texts which communicate relatively simple data. for example, @xcite looked at slot-error-rate in the e2e challenge @xcite , which involved generating short sentences (13 words on average) which communicated 8 attributes. our goal is to develop techniques which can be used to evaluate accuracy in longer texts (hundreds of words) which communicate complex data and possibly insights (eg, trends and best/worst) derived from the source data. this is more challenging task; longer texts can have contextual errors which are rare in short texts, and checking accuracy of insights derived from complex data is harder than checking whether a small number of attributes, for a single entity, are accurately communicated. in this paper we specifically focus on finding accuracy mistakes in english language sports stories. however, we believe the techniques could also be applied to other types of texts produced by data-to-text systems, including financial reports and business intelligence, which are very important in commercial data-to-text applications @xcite . finding accuracy mistakes in a 300-word sports story using our methodology costs on the order of us$30 in mechanical turk worker payments and amazon fees, plus 30 minutes of experimenter time. workers were screened with a qualification task. we intend our methodology to be a high quality gold standard for accuracy evaluation, and encourage other researchers to find alternative and cheaper evaluation techniques which correlate well with the gold-standard presented here. in other words, researchers developing metrics for measuring accuracy can compare the results of their metrics with our gold-standard accuracy evaluation, and use this comparison to validate their metrics and assess how effective their metrics are at measuring accuracy. figure 1 shows example sentences from a sports story annotated by our methodology. the text is an example, constructed from fragments from the output of different systems, with some manual adjustment to keep the example simple. the materials used to perform the evaluation described below, as well as the small corpus of 21 accuracy-annotated sports stories has been released on github foot_0 . ## related work nlg systems can be evaluated using either automatic metrics or human evaluation @xcite . automatic metrics such as bleu are not very meaningful in nlg @xcite , especially when assessing accuracy @xcite . even in machine translation, bleu and related metrics are not meaningful unless the differences in metric scores is quite large, much larger than reported in most academic papers @xcite . human evaluation of nlg systems is usually done using likert scales or ratings (van der @xcite . in the context of evaluating accuracy, human evaluators are usually asked to assess the overall accuracy of a generated text @xcite , or to compare two texts and say which text is overall most accurate @xcite . the pyramid method @xcite ) in text summarisation is a complex technique for evaluating the quality of a summary from a content perspective. it originally required substantial human input, but recently there have been attempts to automate pyramid analysis @xcite . however, pyramid focuses on checking whether expected content is present, not finding mistakes in unexpected content. in the context of evaluating computer-generated sports stories, @xcite showed sentences (not complete stories) to human subjects, and asked the subjects to count how many facts in the sentence were supported by game data and how many contradicted the game data. these results were then compared to metrics based on information extraction techniques. this was repeated by @xcite and extended to other domains by @xcite . another metric which semantically analyses generated text and compares this to the source data is spice @xcite , which uses this approach to evaluate the quality of computergenerated image captions, accuracy-checking is also an issue in fact checking and verification. the fever workshops and shared tasks @xcite asked participants to develop techniques to identify factual the memphis grizzlies (5-2) defeated the phoenix suns (3 -2) monday 102-91 at the talking stick resort arena in phoenix. the grizzlies had a strong first half where they out-scored the suns 59-42. marc gasol scored 18 points, leading the grizzlies. isaiah thomas added 15 points, he is averaging 19 points on the season so far. the suns' next game will be on the road against the boston celtics on friday. list of errors: ## methodology in summary, our approach is to ask multiple annotators to identify specific errors in a text, and categorise the errors into one of a small number of types. we also ask annotators to provide corrections and optionally explanations of the error. an example is shown in figure 1 . we then integrate the annotations into a single gold standard, based on majority opinion of our annotators. the methodology described below has been refined based on results of pilot annotation exercises performed with a different group of participants. ## experiment we used our methodology to annotate a small corpus of sports stories. ## future work we would like to improve the annotation scheme, and ideally also create a proper annotation tool. in addition to speeding up annotation, such a tool could encourage annotators to follow guidelines such as annotating incorrect days-of-week as name errors. we would also like to annotate human authored texts using our methodology in order to provide a topline for nlg systems. we are also planning to run a shared task for accuracy evaluation, where researchers propose faster and cheaper ways of finding accuracy mistakes (either via a human protocol or with a metric), and these techniques are evaluated by seeing how closely they correlate with the gold-standard accuracy evaluation described in this paper. the shared task is described in a companion paper @xcite . ## conclusion texts generated by nlg systems need to be accurate, but current neural nlg systems often generate texts with many mistakes (