Referring to what you know and do not know: Making Referring Expression Generation Models Generalize To Unseen Entities (2020)

Abstract: Data-to-text Natural Language Generation (NLG) is the computational process of generating natural language in the form of text or voice from non-linguistic data. A core micro-planning task within NLG is referring expression generation (REG), which aims to automatically generate noun phrases to refer to entities mentioned as discourse unfolds. A limitation of novel REG models is not being able to generate referring expressions to entities not encountered during the training process. To solve this problem, we propose two extensions to NeuralREG, a state-of-the-art encoder-decoder REG model. The first is a copy mechanism, whereas the second consists of representing the gender and type of the referent as inputs to the model. Drawing on the results of automatic and human evaluation as well as an ablation study using the WebNLG corpus, we contend that our proposal contributes to the generation of more meaningful referring expressions to unseen entities than the original system and related work. Code and all produced data are publicly available.

## introduction data-to-text natural language generation (nlg) is the computational process of generating natural language in the form of text or voice from non-linguistic data. a traditional micro-planning task within the pipeline data-to-text architecture is referring expression generation (reg) @xcite , which aims to automatically generate appropriate noun phrases (e.g., the mathematician ada lovelace) to refer to entities (e.g., ada lovelace) mentioned as discourse unfolds (e.g., " traditionally, reg systems produce references to discourse entities in two explicit steps. first, they decide on the referential form, i.e., choosing whether a referring expression should be a pronoun (she), a proper name (ada lovelace), a description (the mathematician), etc. once the choice is made, such systems textually realize the referring expression based on the chosen referential form and discourse context. if the first step selects a proper name as the form to refer to ada lovelace for instance, the ensuing step is responsible for deciding, among ada, ada lovelace, or another text realization of a proper name, i.e., the one that is the most appropriate referring expression to that entity in a given discourse context. with the advent of large amounts of data, reg systems have undergone a significant change in their architecture. from being rule-based modular, they have become data-driven end-to-end systems that aim to perform the choice of referential form and surface realization jointly. an example of these more integrated approaches is neuralreg @xcite , an end-to-end neural reg model that produces referring expressions deciding on form and content jointly based on representations of the referent and its surrounding context. although neuralreg is able to generate adequate referring expressions to discourse entities already seen during the training phase, the model does not generalize to unseen ones, i.e., it can not generate referring expressions to entities which were not seen during its training. this study aims to fill this gap by proposing two extensions to the model's original architecture. the first is a copy mechanism, which may decide at each decoding timestep whether the next token of the referring expression should be generated from the output vocabulary or copied from the input representation of the target entity. we thereby hypothesize that the model will be able to generate a token from the vocabulary for seen entities and to copy tokens from the input representation in the case of unseen ones. the second extension consists of representing the gender and type of the entity as input to the model. such information can be easily extracted from the semantic web and may help the model to generate pronominal (e.g., she) and descriptive (e.g., the country) referring expressions to unseen entities. to evaluate our approach, we conducted experiments relying on a delexicalized version @xcite of the webnlg corpus @xcite . we first compare our proposal with the original neuralreg and other related approaches as profilereg @xcite . second, to assess the quality of the texts generated by our model, we conducted a supplementary evaluation with human judges. next, we follow the rationale of ablation studies to analyze the importance of each feature in our model within the process of referring expression generation. finally, we discuss some advantages of the introduced features and how they interact to improve accuracy, variety, and generalization. ## related work given an entity to be referred to in a particular context, traditional reg methods have addressed this task in two steps. the first one concerns the choice of referential form, i.e., deciding whether the target reference is more likely to be a proper name (belo horizonte), a description (the city), a pronoun (it), or another referential form. regarding this step, @xcite suggested to always choose a full proper name as the first reference to a particular entity in a given context, whereas pronouns may be used for its subsequent references if there is no other entity with the same person, gender and number in-between the target reference and its antecedents. @xcite proposed a naive bayes method, which is able to non-deterministically choose a referential form to a particular reference. the model's choice is conditioned upon discourse features which studies in psycholinguistics have shown to impact this choice, such as grammatical position, givenness and recency of the target reference. once the referential form is chosen, the second step of traditional reg models focuses on the surface realization of the reference. most part of the literature on this step focuses on the generation of descriptions @xcite although some studies have approached the generation of proper names @xcite @xcite . in contrast to previous proposals that have focused on selecting referential form or referential content, castro @xcite proposed an end-to-end approach: neuralreg, a referring expression generator able to perform the choice of referential form and the surface realization in an end-to-end style using a neural encoder-decoder architecture. given an entity to be referred to in a particular textual context, the approach first encodes the entity identifier and the text prior (pre-context) and subsequent to the reference (post-context) to later decode this representation into an appropriate referring expression using attention @xcite . although @xcite can generate suitable referring expressions to entities seen during training, it presents certain problems when referring to unseen ones. to overcome this limitation, @xcite presented a profile based model. their solution uses information from both profile foot_2 (i.e., information retrieved from the entity's wikipedia page) and context (pre-and post-contexts jointly) to generate suitable references to unseen entities. the authors conclude that their approach is more successful to determine the most suitable referring expression to a particular entity. in contrast to @xcite solution, in order to address the limitations of dealing with unseen relations and entities, our proposal uses a combination of a copy mechanism together with rep- triples subject predicate object adenan satem birthplace japanese occupation of british borneo abdul taib mahmud successor adenan satem abdul taib mahmud residence sarawak abdul taib mahmud party "barisan raayat jati sarawak" corresponding text adenan satem was born in japanese occupied british borneo. his successor was abdul taib mahmud, who, resides in sarawak and is a member of the "barisan raáyat jati sarawak" party. tag entity/constant referring expression bridge-1 adenan satem adenan satem patient-1 japanese occupation of british borneo japanese occupied british borneo bridge-1 adenan satem his agent-1 abdul taib mahmud abdul taib mahmud patient-2 sarawak sarawak patient-3 "barisan raayat jati sarawak" barisan ra'ayat jati sarawak template bridge-1 was born in patient-1. bridge-1 successor was agent-1, who, resides in patient-2 and is a member of the patient-3 party. ## data we evaluated our proposal based on an enriched version @xcite of the webnlg corpus @xcite . the original resource is a parallel corpus with sets of rdf (resource description framework) triples and their corresponding verbalizations. each rdf triple set consists of subject-predicate-object (e.g., adenan satem | birthplace | japanese occupation of british borneo), which is illustrated in the webnlg dataset is an nlg benchmark that differs from other datasets @xcite due to its data diversity in terms of attributes, patterns, and shapes (i.e., rdf tree shapes from dbpedia). the corpus contains 25,298 english texts verbalizing sets of 1 to 7 rdf triples in 15 different domains. the dataset has five domains exclusive to the test set, providing adequate means to evaluate our model's performance regarding the generation of referring expressions to unseen entities. we used an enriched version of the webnlg corpus obtained by a delexicalization process (i.e., mapping each entity to a generic tag and later replacing their corresponding referring expressions in discourse with these tags) which was created by castro @xcite . to train and evaluate our approach, we have a pre-processing stage where we extract a collection of referring expression entries from the enriched version of webnlg @xcite . this stage is performed once, where we map the webnlg corpus information as the basis to obtain external information without adding new features nor changing the webnlg structure. we avoid a possible impact on the evaluation results since all entities are available on dbpedia. each entry consists of a wikipedia id, i.e., a target entity (adenan satem), a truecased tokenized referring expression (adenan satem or his), and lowercased tokenized pre-(adenan satem was born in) and post-contexts (adenan satem successor was abdul taib mahmud, who, resides in sarawak and is a member of the barisan raayat jati sarawak party.), indicating the surrounding context of the target reference. ## model our approach was based on neuralreg @xcite and aims to generate a referring expression @xmath0, y 2 , ..., y n } with n tokens to refer to a target entity, given the textual context prior to the reference m } with m tokens (e.g., pre-context) and subsequent to the reference ## human evaluation to assess the quality of the texts generated by our proposal and the three baselines, we conducted a supplementary evaluation with human judges. method two applied linguists were recruited to rate the texts. they are proficient in english and have over 20 years' expertise as translators and language advisers. we selected 75 instances of the delexicalized version of the webnlg corpus, considering a unique occurrence for each combination between the number of triples (ranging from 1 to 7) and domain (10 seen and 5 unseen ones). after selecting the set of triples, we collected the corresponding produced versions of each investigated model introduced in this study (our proposal and three baselines). finally, we randomly ordered the final trial set of (4 × 75 =) 300 sentences to decrease the bias of having the 4 generated texts together during the evaluation. the performed evaluation followed the best practices suggested by @xcite and the guidelines in @xcite regarding human evaluations of nlg systems. for instance, we results ## ablation study we also performed an ablation study in order to analyze the performance of the different features used by our proposal. method we evaluated the copy mechanism, pre-and post-contexts, as well as gender and type embeddings in order to determine which feature best influences the model. the performance of every single feature was analyzed by running the model without it and measuring loss, according to the referring expression accuracy metric in the test part of the data, considering all entities as well as only seen and only unseen ones. when removing the copy mechanism, the model has a similar performance to the original neuralreg though with the target entity also represented by the entity embeddings for gender and type. results ## discussion this study set out to address a limitation of neuralreg, a state-of-the-art encoder-decoder referring expression generation system, which is its failure to generate references to entities not previously seen during its training. to solve the problem, we proposed two extensions to the original approach: a copy mechanism and using a multi-token representation for the referent as well as its gender and type. considering pre-and post-contexts where an entity should be referred to and information about the entity's gender and type, at each decoding step our model decides whether the next token of the referring expression should be generated from the output vocabulary or copied from the multi-token input representation of the entity. although our approach set out to improve the generation of referring expressions to unseen entities only, an automatic evaluation shows that it presents competitive results regarding all the models when comparing overall performance and also for seen entities. regarding generation of pronouns and references to unseen entities, our model outperforms profilereg and onlynames. furthermore, human evaluation, conducted to rate the automatically generated sentences showed that our model achieved the best results regarding grammaticality. regarding fluency and semantic adequacy, neuralreg+copy and onlynames presented similar results as shown in surrounding context pre-and post-contexts seem to perform different roles when used as input features to generate referring expressions. based on our ablation analysis, pre-context plays a crucial role, being ranked the most important feature when generating referring expressions to seen entities and third to unseen ones. on the other hand, post-context seems to have a slight contribution only for the generation of references to unseen entities. in fact, when not used, the approach performs better for generation of referring expressions to seen entities. copy mechanism among the input features, the copy mechanism proved an essential feature of the model. its importance was supported by the results in the ablation analysis, which pointed to this feature as the most important for the generation of referring expressions to unseen entities. this confirms the copy mechanism to be a productive addition to neuralreg in order to make it able to work with entities not seen during training. gender and type entity representations besides the copy mechanism, we sought to make neural-reg generalize to unseen entities by feeding it with embedding representations of the referent's gender and type. among the motivations to use these features, we considered how easy it is to access this information in the semantic web, since entities in webnlg are represented by their uris. second, we hypothesized that such representations would allow the model to generate pronominal and descriptive referring expressions to unseen entities. to some extent, the pronominal reference his to the unseen entity adenan satem produced by our approach and depicted in future work although our two proposed extensions allowed neuralreg to perform better when compared to its previous version and also generate superior referring expressions to unseen entities, only-names performed slightly better than our approach in the generation of referring expressions to this kind of entities as well as produced similar results to our model during human evaluation. we assume that part of this result is related to known issues in semantic neural models, such as hallucination, which could influence both automatic and human evaluation results, in particular for unseen entities. to fix this issue, we aim to investigate the generation of synthetic referring expression data to augment the training data and better tune our approaches. results also show that profilereg outperformed our model in the generation of pronouns. we hypothesize this result as an impact of incorrect gender and type information for some entities extracted from dbpedia. for instance, the entity bbc in dbpedia foot_5 is also considered of the type person, leading to the generation of inaccurate descriptions (the person) and pronominal (she or he) outputs. in future work, we aim to manually inspect all type and gender information extracted from dbpedia in order to avoid errors. additionally, to generate better pronominal referring expressions, we will enhance our approach by using the "profile" computed by the profilereg model.