ENT - DESC : Entity Description Generation by Exploring Knowledge Graph (2020)

Abstract: Previous works on knowledge-to-text generation take as input a few RDF triples or key-value pairs conveying the knowledge of some entities to generate a natural language description. Existing datasets, such as WIKIBIO, WebNLG, and E2E, basically have a good alignment between an input triple/pair set and its output text. However, in practice, the input knowledge could be more than enough, since the output description may only cover the most significant knowledge. In this paper, we introduce a large-scale and challenging dataset to facilitate the study of such a practical scenario in KG-to-text. Our dataset involves retrieving abundant knowledge of various types of main entities from a large knowledge graph (KG), which makes the current graph-to-sequence models severely suffer from the problems of information loss and parameter explosion while generating the descriptions. We address these challenges by proposing a multi-graph structure that is able to represent the original graph information more comprehensively. Furthermore, we also incorporate aggregation methods that learn to extract the rich graph information. Extensive experiments demonstrate the effectiveness of our model architecture.

## introduction kg-to-text generation, automatically converting knowledge into comprehensive natural language, is an important task in natural language processing (nlp) and user interaction studies @xcite . specifically, the task takes as input some structured knowledge, such as resource description framework (rdf) triples of bruno mars retro style, funk, rhythm and blues, hip hop music, ... peter gene hernandez @xcite , known professionally as bruno mars, is an american singer, songwriter, multi-instrumentalist, record producer, and dancer. he is known for his stage performances, retro showmanship and for performing in a wide range of musical styles, including r&b, funk, pop, soul, reggae, hip hop, and rock. webnlg @xcite , key-value pairs of wikibio @xcite and e2e @xcite , to generate natural text describing the input knowledge. in essence, the task can be formulated as follows: given a main entity, its one-hop attributes/relations (e.g., wikibio and e2e), and/or multi-hop relations (e.g., webnlg), the goal is to generate a text description of the main entity describing its attributes and relations. note that these existing datasets basically have a good alignment between an input knowledge set and its output text. obtaining such data with good alignment could be a laborious and expensive annotation process. more importantly, in practice, the knowledge regarding the main entity could be more than enough, and the description may only cover the most significant knowledge. thereby, the generation model should have such differentiation capability. in this paper, we tackle an entity description generation task by exploring kg in order to work towards more practical problems. specifically, the aim is to generate a description with one or more sentences for a main entity and a few topic-related entities, which is empowered by the knowledge from a kg for a more natural description. in order to facilitate the study, we introduce a new dataset, namely entity-to-description (ent-desc) extracted from wikipedia and wikidata, which contains over 110k instances. each sample is a triplet, containing a set of entities, the explored knowledge from a kg, and the description. figure 1 shows an example to generate the description of the main entity, i.e., bruno mars, given some relevant keywords, i.e., retro style, funk, etc., which are called topic-related entities of bruno mars. we intend to generate the short paragraph below to describe the main entity in compliance with the topic revealed by topic-related entities. for generating accurate descriptions, one challenge is to extract the underlying relations between the main entity and keywords, as well as the peripheral information of the main entity. in our dataset, we use such knowledge revealed in a kg, i.e., the upper right in figure 1 with partially labeled triples. therefore, to some extent, our dataset is a generalization of existing kg-to-text datasets. the knowledge, in the form of triples, regarding the main entity and topic entities is automatically extracted from a kg, and such knowledge could be more than enough and not necessarily useful for generating the output. our dataset is not only more practical but also more challenging due to lack of explicit alignment between the input and the output. therefore, some knowledge is useful for generation, while others might be noise. in such a case that many different relations from the kg are involved, standard graphto-sequence models suffer from the problem of low training speed and parameter explosion, as edges are encoded in the form of parameters. previous work deals with this problem by transforming the original graphs into levi graphs @xcite . however, levi graph transformation only explicitly represents the relations between an original node and its neighbor edges, while the relations between two original nodes are learned implicitly through graph convolutional networks (gcn). therefore, more gcn layers are required to capture such information @xcite . as more gcn layers are being stacked, it suffers from information loss from kg @xcite . in order to address these limitations, we present a multi-graph convolutional networks (mgcn) architecture by introducing multigraph transformation incorporated with an aggregation layer. multi-graph transformation is able to represent the original graph information more accurately, while the aggregation layer learns to extract useful information from the kg. extensive experiments are conducted on both our dataset and benchmark dataset (i.e., webnlg). mgcn outperforms several strong baselines, which demonstrates the effectiveness of our techniques, especially when using fewer gcn layers. our main contributions include: • we construct a large-scale dataset ent-desc for a more practical task of entity description generation by exploring kg. to the best of our knowledge, ent-desc is the largest dataset of kg-to-text generation. researchers also studied how to generate texts from a few given entities or prompts @xcite . however, they did not explore the knowledge from a kg. graph-to-sequence modeling. in recent years, graph convolutional networks (gcn) have been applied to several tasks (e.g., semi-supervised node classification (kipf and welling, 2017), semantic role labeling @xcite and neural machine translation @xcite ) and also achieved state-of-the-art performance on graph-to-sequence modeling. in order to capture more graphical information, @xcite introduced graph attention networks (gats) through stacking a graph attentional layer, but only allowed to learn information from adjacent nodes implicitly without considering a more global contextualization. @xcite then used gcn as the encoder in order to capture more distant information in graphs. since there are usually a large amount of labels for edges in kg, such graph-to-sequence models without graph transformation will incur information loss and parameter explosion. @xcite proposed to transform the graph into levi graph in order to work towards the aforementioned deficiencies, together with gated graph neural network (ggnn) to build graph representation for amr-to-text problem. however, they face some new limitations brought in by levi graph transformation: the entityto-entity information is being ignored in levi transformation, as also mentioned in their paper. afterwards, deeper gcns were stacked @xcite to capture such ignored information implicitly. in contrast, we intend to use fewer gcn layers to capture more global contextualization by explicitly stating all types of graph information with different transformations. ## task description in this paper, we tackle a practical problem of entity description generation by exploring kg. in practice, it is difficult to describe an entity in only a few sentences as there are too many aspects for an entity. now, if we are given a few topic-related entities as topic restrictions to the main entity, the text to be generated could be more concrete, particularly when we are allowed to explore the connections among these entities in kg. as seen in figure 1 , when we are asked to use one or two sentences to introduce "bruno mars" foot_0 , his popular singles will first come into some people's minds, while his music genres might be in other people's first thought. with the introduction of topic-related entities, the description will have some focus. in this case, when topic-related entities, i.e., r&b, hip hop, rock, etc., are provided, we are aware of describing bruno mars in the direction of music styles on top of their basic information. formally, given a set of entities @xmath0, ..., e n } and a kg @xmath1, e), where e 1 is main entity, e 2 , ..., e n are topic-related entities, v is the set of entity nodes and e is the set of directed relation edges. we intend to generate a natural language text @xmath2, y 2 , • • • , y t }. meanwhile, we explore g for useful information to allow a more natural description. here, the kg g can also be written as a set of rdf triples: ## ent-desc dataset to prepare our dataset, we first use nayuki's implementation foot_1 to calculate the pagerank score for more than 9.9 million wikipedia pages. we then extract the categories from wikidata for the top 100k highest scored pages and manually select 90 categories out of the top 200 most frequent ones as the seed categories. the domains of the categories mainly include humans, events, locations and organizations. the entities from these categories are collected as our candidate set of main entities. we further process their associated wikipedia pages for collecting the first paragraphs and entities with hyperlink as topic-related entities. we then search wikidata to gather neighbors of the main entities and 1-hop/2-hop paths between main entities and their associated topic-related entities, which finally results in a dataset consisting of more than 110k entity-text pairs with 3 million triples in the kg. although more-hop paths might be helpful, we limit to 1-hop/2-hop paths for the first study. the comparison of our dataset with webnlg, agenda and e2e is shown in in the comparison of these four datasets, there are some obvious differences. first, our dataset is significantly larger than webnlg, agenda and e2e (i.e., more than twice of their instances). meanwhile, our vocabulary size and numbers of distinct entities/relations are all much larger. second, the average number of input triples per instance is much larger than those of the other two. more importantly, our dataset provides a new genre of data for the task. specifically, webnlg has a strict alignment between input triples and output text, and accordingly, each input triple roughly corresponds to 8 words. agenda is different from webnlg for generating much longer output, namely paper abstracts, with the paper title also given as input. moreover, as observed, quite a portion of text information cannot be directly covered by the input triples. e2e focuses on the restaurant domain with relatively simple inputs, including 77 entities and 8 relations in total. considering the construction details of these 3 datasets, all their input triples provide useful information (i.e., should be used) for generating the output. in contrast, our dataset has a much larger number of input triples, particularly considering the length difference of output texts. lastly, another unique characteristic of our dataset is that not every input triple is useful for generation, which brings in the challenge that a model should be able to distill the helpful part for generating a better output sequence. ## our mgcn model given the explored knowledge, our task can be cast as a problem of generating text from kg. we propose an encoder-decoder architecture with a multi-graph transformation, shown in figure 3 . ## experiments 6.1 experimental settings we implement our mgcn architecture based on @xcite and sockeye toolkit. hidden units and embedding dimensions for both encoder and decoder are fixed at 360. we use adam @xcite with an initial learning rate of 0.0003 and update parameters with a batch size of 16. the training phase is stopped when detecting the convergence of perplexity on the validation set. during decoding, we use beam search with a beam size of 10. all models are run with v100 gpu. we evaluate our models by applying both automatic and human evaluations. for automatic evaluation, we use several common evaluation metrics: bleu @xcite , me-teor @xcite , ter @xcite , rouge 1 , rouge 2 , rouge l @xcite , parent @xcite . we adapt multeval @xcite and py-rouge for resampling and significance test. ## conclusions and future work we present a practical task of generating sentences from relevant entities empowered by kg, and construct a large-scale and challenging dataset ent-desc to facilitate the study of this task. extensive experiments and analysis show the effectiveness of our proposed mgcn model architecture with multiple aggregation methods. in the future, we will explore more informative generation and consider applying mgcn to other nlp tasks for better information extraction and aggregation.