Grounded Keys-to-Text Generation: Towards Factual Open-Ended Generation (2022)

Abstract: Large pre-trained language models have recently enabled open-ended generation frameworks (e.g., prompt-to-text NLG) to tackle a variety of tasks going beyond the traditional data-to-text generation. While this framework is more general, it is under-specified and often leads to a lack of controllability restricting their real-world usage. We propose a new grounded keys-to-text generation task: the task is to generate a factual description about an entity given a set of guiding keys, and grounding passages. To address this task, we introduce a new dataset, called EntDeGen. Inspired by recent QA-based evaluation measures, we propose an automatic metric, MAFE, for factual correctness of generated descriptions. Our EntDescriptor model is equipped with strong rankers to fetch helpful passages and generate entity descriptions. Experimental result shows a good correlation (60.14) between our proposed metric and human judgments of factuality. Our rankers significantly improved the factual correctness of generated descriptions (15.95% and 34.51% relative gains in recall and precision). Finally, our ablation study highlights the benefit of combining keys and groundings.

## introduction converting information to text @xcite has been a cornerstone of nlg research with the goal of improving the accessibility of knowledge to general users. it has found many applications such as generating sport commentaries @xcite , weather forecast @xcite , biographical text @xcite , and dialogue response generation @xcite . the problem has traditionally been formulated as data-to-text generation, to generate an output given structured input such as graph, tables or key-value pairs. however, this formulation is overspecified and does not cover other open-ended scenarios in real-world. recent advances in large pre-trained language models (plms), as well as the general knowledge represented in them, have made it possible to formulate the problem as prompt-totext or outline-to-text @xcite generation. this offers the prospect of making nlg more broadly applicable, as such models allow input to be more parsimonious or ill-defined. however, issues such as lack of controllability and hallucination have lessened the practical applicability of this setting in real-world scenarios. to overcome these issues, we propose a new task, grounded keys-to-text generation, where given a wishlist of keys (without the values) about an entity foot_0 and a set of short grounding passages as a source knowledge, the goal is to generate a factual description. an example is shown in fig. 1 , where the task is to generate a paragraph about "barack obama", in particular about his family and personal life. potential factual keys in this example are "birth date, birthplace, parents, spouse, children", etc. the task also enables a finer-grained control over the types of entities to be included in the output via topical keys such as "hospital, city, law firm" for the example in fig. 1 . finally, pertinent information about the entities needs to be fetched from a set of candidate grounding passages. these passages can be obtained via internet search. our task differs from similar existing tasks, such as data-to-text generation @xcite in that, we presume keys but not values are given. this covers more open-ended scenarios in the real-world where knowledge about entities are not available in detailed structured format, is constantly changing and so have to be fetched on the fly. moreover, this formulation offers control to the user over the generated text. to facilitate research on grounded keys-to-text generation task, we introduce a large-scale and challenging dataset, called entdegen, with about 375k instances. the grounded, factual and longform nature of the task, brings a new challenge, i.e., generating paragraph-level text which is faithful to one or more grounding passages based the provided guiding keys. to address this challenge, we propose entdescriptor equipped with strong ranker to help the model focus on passages that are both relevant to the keys and complementary. we propose two rankers. our contrastive dense ranker is based on embedding-based retrieval systems trained in a contrastive framework. our autoregressive ranker generates a sequence of passage indices autoregressivly by modeling the probability of each passage conditioned on previously generated passages. this ranker is shown to achieve the strongest performance by modeling the joint probability of passages. the factual aspect of generation also calls for a new evaluation metric. inspired by recent factbased evaluation for summarization, we propose an automatic metric, called mafe, to evaluate different aspects of grounded text quality, including relevance and consistency. our contributions are: 2 • we introduce a new controllable entity description generation task which requires aggregating knowledge from multiple grounding passages efficiently. ## dataset: entdegen our dataset collection strategy is based on wikipedia and motivated by the wikitablet dataset @xcite . each wikipedia article a w is composed of multiple sections @xmath0, s 2 , ..., s n }. the title of the wikipedia article is the entity e whose description is to be generated and the text in each section forms a reference (gold) description, r. for example, an article about a football player may contain sections about "introduction", "early life", "club career", and "international career", each forming a separate instance in our dataset. we perform the following steps for each section s i in an article to obtain: factual keys, topical keys, and grounding passages. factual keys. factual keys seek specific knowledge about an entity of interest.for obtaining factual keys, we align key-value pairs in infobox and wikidata with each section s i . for this, we took a distant-supervision approach to estimate the alignment score of each key-value pair with the section using semantic similarity and lexical precision. for semantic similarity, we compute the precision component of bert-score @xcite between the section text and the concatenation of keyvalue pair (key + value). a high value indicates that the key-value pair is semantically relevant to that section. we also measure the rouge-l precision score @xcite between the section text with respect to the concatenation of key-value pair. for each instance in our dataset, we select keys whose key-value bert-score is greater than 0.82, and rouge-l score is greater than 0.25. ## mafe: multi-aspect factuality evaluation our proposed task is to generate a factual description. hence, it is crucial to evaluate the factuality of the generated texts. inspired by recent fact-based evaluation in abstractive summarization @xcite @xcite , we propose to assess the factuality of generation through question answering (qa). we evaluate factuality of a generated description h with respect to (i) factual triples (e, k, v) which are constructed from the entity e, each factual key k and its value v, foot_2 and (ii) reference (gold) description r. in our qa based multi-aspect factualy evaluation (mafe), questions are generated from spans in the reference and factual triples (recall), or the generated output (precision), and are automatically answered using the output, or reference-factual triples. then, the similarity between the predicted answer and the gold answer is used to compute recall and precision. our evaluation framework is illustrated in fig. 2 which accounts for both relevance (recall) and consistency (precision): recall (@xmath1, k, v) and reference r. for this, we generate questions that have gold answers in factual triples and reference using a question generation (qg) module, and obtain answers to these questions from generated output h using a question answering (qa) module. we define recall as the average scores of these answers when compared to the gold answers (computed by an answer matching (am) module). precision (@xmath2, k, v) or reference r. for this, we generate questions from output and obtain answers from factual triples and reference. we define precision as the maximum score between answers predicted from factual triples and reference. next, we describe the 3 modules of the evaluation framework. ## entdescriptor the entdescriptor model needs to fetch relevant passages on the fly to generate a factual description. for this, we equip our entdescriptor model with a passage ranker ( §5.1). given the entity, keys and a set of ranked passages, the descriptor generator ( §5.2) then generates an entity description. ## conclusion we present a practical task of grounded keys-totext generation and construct a large-scale dataset entdegen to facilitate research on this task. experiments show the effectiveness of the proposed rankers to fetch relevant information required to generate a factual description. the human evaluation shows that entdegen poses a challenge to state-of-the-art models in terms of achieving human-level factuality in long-form generation. our proposed dataset and task can also foster further research in the recently emerging retrieval augmented generations models @xcite @xcite -where the retriever and generator components are trained end-to-end.