Evaluation Guidelines to Deal with Implicit Phenomena to Assess Factuality in Data-to-Text Generation (2021)

Abstract: Data-to-text generation systems are trained on large datasets, such as WebNLG, Ro-toWire, E2E or DART. Beyond traditional token-overlap evaluation metrics (BLEU or METEOR), a key concern faced by recent generators is to control the factuality of the generated text with respect to the input data specification. We report on our experience when developing an automatic factuality evaluation system for data-to-text generation that we are testing on WebNLG and E2E data. We aim to prepare gold data annotated manually to identify cases where the text communicates more information than is warranted based on the in-put data (extra) or fails to communicate data that is part of the input (missing). While analyzing reference (data, text) samples, we encountered a range of systematic uncertainties that are related to cases on implicit phenomena in text, and the nature of non-linguistic knowledge we expect to be involved when assessing factuality. We derive from our experience a set of evaluation guidelines to reach high inter-annotator agreement on such cases.

## introduction we investigate how to deal with implicit phenomena in text when assessing whether generated text is faithful to an input data specification. recent data-to-text generation systems are trained on large dataset, such as webnlg @xcite , e2e @xcite , wikibio @xcite , or @xcite . datato-text systems are usually evaluated by comparing the generated text with reference text with metrics such as bleu @xcite , meteor @xcite or bertscore @xcite . yet, recent work has shown that neural generation models risk to create text that is not faithful to the input data specification @xcite , by either introducing content which is not warranted by the input or failing to express some of the content which is part of the input. in fact, studies indicate that the training datasets also suffer from problematic alignment between data and text (e.g., @xcite and @xcite ), because large-scale data collection is complex. in response, new evaluation metrics are being developed to assess the factuality of text with respect to the input data. @xcite attempts to measure factuality by extracting data from the generated text using openie techniques and comparing it with the original data. @xcite exploit a roberta-based nli system to check bidirectional entailment relation between generated text and input data. @xcite operate without reference text and use a qa-based evaluation method to assess whether the text answers questions that are generated on the basis of the input data. in this paper, we investigate what it means for text to be faithful to data by revisiting human guidelines, specifically related to implicit phenomena in text. while manually assessing the factuality of text generated by a generator we developed on the webnlg and e2e datasets, we encountered systematic uncertainties in deciding whether text was missing or unwarranted given the data. we faced such uncertainties in more than half of the cases we analyzed. we provide here a list of such cases that we categorize according to the type of implicit phenomenon which triggers uncertainty. our main contribution is a set of guidelines for human annotation of data-to-text datasets in terms of semantic alignment: does the text convey all the data in the input, and does it introduce unwarranted #@xmath0,_sri_lanka #@xmath1,_sri_lanka #@xmath2,_sri_lanka<@xmath3, but in the eastern province it is austin fernando. this is where the ampara hospital is located in ampara district. or contradictory content. ## content conveyed implicitly by text vs. data we report on observations gathered during a larger effort: to study the factuality of generated text vs. a data input specification, we sample (data, text) pairs from existing datasets, and synthesize new noisy pairs where we either add a predicate to the data side, remove one, or alter an existing predicate (e.g., transform a triplet region(ampara hospital, ampara district) into region(ampara hospital, northern province)). given these pairs (both original and synthesized), we manually annotate the pairs as either: reliable (text faithfully matches the data), missing (text fails to cover part of the data), extra (text hallucinates content which is not part of the data) and perturbation (a combination of missing and extra, meaning some of the content conveyed by the text was altered with respect to the input data). while annotating this data, we identified systematic cases of uncertainty: we categorize six cases where the relation between generated text and the input data is uncertain, either because the text conveys content in an implicit way or because the input data entails additional facts. we give examples taken from reference text in the webnlg dataset. we then provide statistics on the prevalence of these cases of vague semantic relation. ## discussion the review of the examples above illustrates the complexity of determining whether text conveys data in a faithful manner. in the same way as text conveys implicit content, we observe that the small data snippets currently used as input to data to text systems do not have precise semantics: are the relations collective, transitive, symmetric, time is not specified, entities are referenced with non-arbitrary labels which are interpreted in vague manner. as a consequence, we suggest that we should consider the task of aligning text with data as a text to text alignment, which demands the annotator to exploit world knowledge and common sense. we follow in this the approach of @xcite who cast the task of factuality checking as bidirectional textual entailment and rebuffel et al. ( 2021 ) who view it as question-answering. our contribution is to translate this approach into more precise guidelines for human evaluation, taking into account aspects of implicit communication in language. we have prepared a set of guidelines answering the uncertainties listed above on the basis of this general approach. based on these guidelines, we have manually annotated 200 samples from webnlg with two annotators. we found a high rate of samples in the reference data which suffer from poor alignment, as was reported in previous work for a variety of datasets (e.g., @xcite ). we also find low alignment between our manual annotation and the automatic assessment figure 4 : expression of implicit knowledge in webnlg: the fact that turner is a musician is not explicitly stated in the input the antares rocket is comparable to the delta ii, which originates from the united states. the launch site of the antares was the mid atlantic regional spaceport launch pad 0, which is also associated with the rocket minotaur iv. figure 5 : redundant data in webnlg input: t1 is implied by t4 and the form of the launch0 label tool provided by @xcite . this indicates the task of assessing the semantic faithfulness of generated text in data to text remains challenging, both manually and automatically.