DESCGEN : A Distantly Supervised Datasetfor Generating Entity Descriptions (2021)

Abstract: Short textual descriptions of entities provide summaries of their key attributes and have been shown to be useful sources of background knowledge for tasks such as entity linking and question answering. However, generating entity descriptions, especially for new and long-tail entities, can be challenging since relevant information is often scattered across multiple sources with varied content and style. We introduce DESCGEN: given mentions spread over multiple documents, the goal is to generate an entity summary description. DESCGEN consists of 37K entity descriptions from Wikipedia and Fandom, each paired with nine evidence documents on average. The documents were collected using a combination of entity linking and hyperlinks into the entity pages, which together provide high-quality distant supervision. Compared to other multi-document summarization tasks, our task is entity-centric, more abstractive, and covers a wide range of domains. We also propose a two-stage extract-then-generate baseline and show that there exists a large gap (19.9% in ROUGE-L) between state-of-art models and human performance, suggesting that the data will support significant future work.

## introduction entity knowledge has been shown to play an important role in various applications including language modeling @xcite , open-domain question answering @xcite , and dialogue generation @xcite . recent studies suggest that such entity knowledge can be provided by simple textual descriptions @xcite , which can be incorporated to improve downstream task performance (nie et al., 2018; logeswaran 1 data and code available at github.com/swj0419/descgen doc 1 ...are bitcoins, then, really worth anything? according to carl menger's subjective theory of value, they are worth whatever individuals choose to believe they are worth. it is clear that many individuals value this new medium of exchange highly... doc 2 ...the austrian school of economics has its roots outside of austria -particularly in the french economists jean baptiste say and claude-frederic bastiat. the austrian school proper began with carl menger, who challenged the british labor theory of value. to learn more about austrian economics go to the website of the ludwig von mises institute... doc 3 ...karl menger was born on january 13, 1902, in vienna. his father was the famous austrian economist carl menger who was one of the founders of marginal utility theory.... entity description carl menger @xcite was an austrian economist and the founder of the austrian school of economics. he contributed to the development of the marginal utility theory and to the formulation of a subjective theory of value. descgen contains 37k entity descriptions extracted from wikipedia and fandom 2 . fandom allows us to capture the key challenge of generating descriptions for emerging entities that are not in wikipedia because they are less popular or have just been introduced to the public. to obtain source documents of the entities, we collect web documents and news articles where entity mentions are linked using web hyperlinks or an entity linker. our dataset is distantly supervised in that these heuristically collected documents are not guaranteed to contain all the facts required to generate the description-as would be seen for natural text collections describing emerging entities. we also carefully annotate a subset of 1,000 examples to support more reliable evaluation (see unlike multi-document summarization that makes the assumption that a set of documents to be summarized are written on the same topic @xcite , descgen only assumes that source documents mention the entity. in contrast to an existing entity summarization benchmark @xcite , descgen is more abstractive and better approximates challenges faced when describing new entities. section 4.4 provides more details on these comparisons. overall, our documents for generating a description can cover a much wider range of topics as well as text genres, including news, blog posts, and scientific articles. for instance, the documents 1 and 2 mentioning carl menger in figure 13 discuss topics on bitcoins and the austrian school of economics. finally, we also propose a two-stage method that first extracts salient sentences relevant to the entity and then abstracts them into a description. we test a range of models to establish baseline results with both automatic and human evaluation. the best model based on bart @xcite achieves 28.2% in the rouge-l f measure with a significant gap compared to the human performance 48.1%, suggesting there was great room for future improvement. in summary, our contributions include: ## related work existing entity description generation task and dataset previous works @xcite @xcite ) mainly take as input some structured data such as knowledge graphs to generate entity descriptions. however, knowledge graphs, often mined from text corpora, are overwhelmingly incomplete on real-world entities and may not be updated in real-time @xcite . therefore, we focus on generating descriptions from natural language sources such as web texts and news because they are often primary sources for entities and have better coverage of entities across multiple domains. descgen is most related to wikisum, a recent dataset for generating wikipedia summaries from textual sources @xcite . wikisum source documents primarily come from high-quality articles cited in the wikipedia pages which makes their data more extractive (section 4.4). in contrast, we collect our source documents heuristically using web texts and news, providing a better proxy for emerging entities where high-quality citation sources may not be available. in addition, their evaluation is conducted only on distantly supervised test data. however, our experiments demonstrate that manually annotated data allows for much better evaluation of model performance ( multi-document summarization aims to condense a cluster of thematically-related documents into a short and informative summary. a wide range of multi-document summarization datasets have been built for the document understanding and text analysis conferences @xcite , news @xcite , events @xcite and wikipedia summaries @xcite . recent work has studied both extractive @xcite @xcite and abstractive summarization @xcite @xcite . however, existing datasets typically are not entity focused and assume the input documents are at least loosely centered around a coherent topic or event. wikipedia generation our work is also related to research on generating wikipedia articles. for instance, sauper and @xcite learn to build content templates using an integer linear program to generate full articles. similarly, @xcite generate wikipedia pages by building a topic classifier to assign web retrieved contents into relevant sections. we focus on a different taskgenerating a short text description that can identify and best summarize an entity. ## dataset collection task definition given a collection of documents @xmath0...n} with mentions linked to the same entity e, the goal is to generate a description of e. for example, ## dataset analysis an analysis of the data shows that descgen contains a high proportion of emerging entities from diverse domains, and is more extractive compared to other multi-document summarization datasets. ## baselines in this section, we introduce several new baseline methods, building on state-of-the-art pre-trained models. the input documents can be long (section 8), making it computationally infeasible to train end-to-end models. we instead introduce a pipelined approach to generate an entity description in two stages. in the first extractive stage, a selector is used to identify representative sentences relevant to the entity from multiple source documents. in the second abstractive stage, a neural generation model is used to fuse the selected sentences to a description of the entity. we compare a number of different approaches for each stage, as summarized in the subsections below. ## experiments 6.1 evaluation metrics following other summarization tasks, we evaluate the quality of generated descriptions by rouge extract. abstract. ## analysis in this section, we perform qualitative and quantitative analysis of baseline results to better understand strengths and weaknesses of models, and hypothesize avenues for future work. ## conclusion in this work, we introduce descgen, a new dataset for generating entity descriptions from mentions. descgen contains 37k pairs of entity descriptions from wikipedia and fandom, and 481k automatically gathered source documents based on distant supervision. we also present a clean human-authored subset of 1,000 pairs for test. we show that, as compared to existing benchmarks, descgen requires more abstractive summaries, which we argue better approximate the challenge of describing emerging entities. we also show that the performance of state-of-art models is far from human levels, suggesting that our task remains a significant challenge with room for improvement. our study points to an interesting research direction on modeling entity knowledge from contexts. we hope it will facilitate future work on incorporating entity knowledge into downstream tasks and generating descriptions for emerging entities. doc 1 ...it sometimes gets confusing in the global village , where technology, finance, cross-cultural interactions, and expanding ethnic diasporas are tearing apart the relationship between borders and making multiple identities possible. hence, ang lee is a taiwanese artist who directs american films, but he is also an american film director of chinese movies. as a member of the sinosphere, enlarged by fifty million overseas chinese, ang is not only a creative individual who makes our world more interesting and prosperous. he also helps to bridge between nations and cultures and to produce a sino-american synergy that is more conducive to peace than a contingency of chinese and u.s. diplomats... doc 2 ...the life of pi. one of the most interesting film adaptations set for release in 2012 is brokeback mountain fame. suraj sharma, who has no previous acting experience, will play the central character, piscine patel. based on the novel by yann martel, it is being brought to the big screen by ang lee... doc 3 comic character hulk is dr. bruce banner, who becomes a green monster with powerful strength after an experiment went bad, or well, depending on who you ask. in 2003, director ang lee's film hulk brought this character to the big screen, but was poorly received by hulk's fans... wikipedia description ang lee, @xcite , is an taiwan-born film director who transitioned from directing chinese films to major english-language productions. human-authored description ang lee is a taiwanese director who directs american and chinese films. he is a director of the life of pi and hulk and regarded as second new wave of taiwanese directors. bart ang lee is a taiwanese film director and screenwriter.