Entity-Based Semantic Adequacy for Data-to-Text Generation (2021)

Abstract: While powerful pre-trained language models have improved the fluency of text generation models, semantic adequacy -the ability to generate text that is semantically faithful to the input- remains an unsolved issue. In this paper, we introduce a novel automatic evaluation metric, Entity-Based Semantic Adequacy, which can be used to assess to what extent generation models that verbalise RDF (Resource Description Framework) graphs produce text that contains mentions of the entities occurring in the RDF input. This is important as RDF subject and object entities make up 2/3 of the input. We use our metric to compare 25 models from the WebNLG Shared Tasks and we examine correlation with results from human evaluations of semantic adequacy. We show that while our metric correlates with human evaluation scores, this correlation varies with the specifics of the human evaluation setup. This suggests that in order to measure the entity-based adequacy of generated texts, an automatic metric such as the one proposed here might be more reliable, as less subjective and more focused on correct verbalisation of the input, than human evaluation measures.

## introduction with the introduction of pretrained models, the fluency of text generation systems has improved. however, semantic adequacy (faithfulness to the input) remains an unsolved issue. it remains difficult to ensure that the generated text faithfully captures the input @xcite . in this paper, we focus on semantic adequacy for rdf-verbalisers i.e., models such as those submitted to the webnlg 2017 and 2020 shared tasks @xcite which map an rdf graph to a text verbalising the content of that graph. in this case, the input to natural language generation (nlg) is a set of triples of the form (e1, p, e2) where e1, e2 are rdf entities and p is a property. rdf triplestores are used in particular to model semantic web data and their verbalisation aims at making the information from these knowledge-bases easily accessible to users. as exemplified in figure 1 , one necessary condition for the generated text to be semantically adequate is that all entities present in the input should be mentioned at least once in the output. we refer to this requirement as entity-based semantic adequacy (esa for short). esa offers one way of formalising the requirement that the output of a generator should reflect the information in the input. thus, its significance extends beyond the specific problem domain of rdf verbalisation, though the latter provides a useful testcase. we make the following contributions. we devise metrics which assess to what extent a text verbalising an rdf graph respects entitybased semantic adequacy. these metrics rely on an algorithm designed to automatically detect whether an entity present in the input graph has a corresponding mention in the output text. we evaluate this algorithm on a corpus of 25,173 (rdf, text) pairs with manually annotated entity mentions from castro @xcite and show that our algorithm has a recall of 0.74 and a precision of 0.75. we apply these metrics to the output of 25 rdf verbalisers developed for the webnlg 2017 and 2020 challenges and show that some of the systems which rank highest in terms of bleu scores actually rank in the lower half with respect to entitybased semantic adequacy. this indicates that esa is measuring a different quality from that measured by surface-based metrics such as bleu. we compute correlation between our metrics and both automatic and human evaluation scores collected by the webnlg organisers for these 25 models. we find a stronger correlation with human metrics related to semantic adequacy than with automatic metrics. among the automatic metrics, 1 (short) output text liselotte grschebina is a german national who was born in the german empire and has a total area of 20769100000. 0. rdf input liselotte_grschebina nationality israel israel areatotal 20769100000.0 israel officiallanguage modern_standard_arabic liselotte_grschebina birthplace german_empire liselotte_grschebina training school_of_applied_arts_in_stuttgart 2 (hal) output text born in the kingdom of england in 1726-01-01, and living in india, on the 18th of july, 1776, the country is the birth place of joh davutoglu. rdf input lady_anne_monson birthplace darlington lady_anne_monson birthdate 1726-01-01 lady_anne_monson deathdate 1776-02-18 lady_anne_monson birthplace kingdom_of_england lady_anne_monson residence india 3 (deg) output text the distributor of the distributor of the distributor of the distributor of the distributor of the distribution of the distribution of the distribution of the dish, roadside attrón, is tom botta, who starred in the preparation of the tennis katzman. rdf input super_capers editing stacy_katzman super_capers starring michael_rooker super_capers starring tom_sizemore super_capers language english_language super_capers distributor roadside_attractions figure 1: examples of outputs with low entity-based semantic adequacy. rdf input entities that are missing in the text are underlined (short: the short output fails to mention all input entities, deg: degenerate output, hal: the text hallucinates entities not present in the input and omits to mention others) ## defining e-based semantic adequacy we assume a corpus of (r, t ) instances where r is an rdf graph (a set of rdf triples) and t is a text verbalising that graph. rdf triples are of the form (s, p, o) where p is a binary relation holding between a subject (s) and an object (o) foot_0 . we use the term "entity" to refer to both rdf triple subjects and objects and we write e r for the set of rdf entities occurring in rdf graph r. ## computing e-based semantic adequacy the metrics introduced in the previous section rely on being able to determine which entities in the rdf input have a matching mention in the corresponding text. we present an algorithm for entity mention detection and we report on an evaluation of that algorithm using a dataset of 25,173 (rdf graph, text) pairs where entity mentions have been manually annotated. ## evaluating rdf-to-text generation models finally, we show some example outputs illustrating different ways in which a generated text might have low entity-based semantic adequacy. ## nlg. zooming in on the more state-ofthe-art 2020 models, we find that out of a total of 1779 texts and 16 model outputs, the average esi c 1 score is 17% and the median 10%. in other words, on average, models fail to mention at least one entity 17% of the time. there are strong differences between the models however. the rule-based models (rali, baseline-2017, dangnt-sgu, baseline-2020) have low esi c 1 . this is unsurprising as such models can integrate lexicons mapping rdf entities to natural language mentions. interestingly, among the other five models with an esi c 1 less than 11%, four are bilingual neural nlg models i.e., models which were trained to transform rdf data not only in english but also in russian. ## correlation with human and automatic metrics we study the correlation of our esa i metric with the human and automatic metrics used in the webnlg challenges. ## conclusion rdf stores have become increasingly popular as a means to make knowledge available on the web @xcite . we propose an automatic metric for assessing the entity-based semantic adequacy of rdf verbalisers and show that it is effective in highlighting semantic inadequacy even for state-ofthe-art models with high bleu scores. we further show that models detected by this metric as having low entity-based semantic adequacy can still have high scores on surface-based metrics, and that while esa correlates with human scores on semantic criteria, it may in fact be more reliable as a means of detecting low performing models than human-based evaluation protocols, which tend to be subjective.