Modeling Global and Local Node Contexts for Text Generation from Knowledge Graphs (2020)

Abstract: Recent graph-to-text models generate text from graph-based data using either global or local aggregation to learn node representations. Global node encoding allows explicit communication between two distant nodes, thereby neglecting graph topology as all nodes are directly connected. In contrast, local node encoding considers the relations between neighbor nodes capturing the graph structure, but it can fail to capture long-range relations. In this work, we gather both encoding strategies, proposing novel neural models that encode an input graph combining both global and local node contexts, in order to learn better contextualized node embeddings. In our experiments, we demonstrate that our approaches lead to significant improvements on two graph-to-text datasets achieving BLEU scores of 18.01 on the AGENDA dataset, and 63.69 on the WebNLG dataset for seen categories, outperforming state-of-the-art models by 3.7 and 3.1 points, respectively.1

## introduction graph-to-text generation refers to the task of generating natural language text from input graph structures, which can be semantic representations @xcite or knowledge graphs (kgs) @xcite . whereas most recent work @xcite @xcite focuses on generating sentences, a more challenging and interesting scenario emerges when the goal is to generate multisentence texts. in this context, in addition to sentence generation, document planning needs to be handled: the input needs to be mapped into several sentences; sentences need to be ordered and connected using appropriate discourse markers; and inter-sentential anaphora and ellipsis may need to be generated to avoid repetition. in this paper, we focus on generating texts rather than sentences where the output are short texts @xcite or paragraphs @xcite . a key issue in neural graph-to-text generation is how to encode the input graphs. the basic idea is to incrementally compute node representations by aggregating structural context information. to this end, two main approaches have been proposed: (i) models based on local node aggregation, usually built upon graph neural networks (gnns) @xcite and (ii) models that leverage global node aggregation. systems that adopt global encoding strategies are typically based on transformers @xcite , using self-attention to compute a node representation based on all nodes in the graph. this approach enjoys the advantage of a large node context range, but neglects the graph topology by effectively treating every node as being connected to all the others in the graph. in contrast, models based on local aggregation learn the representation of each node based on its adjacent nodes as defined in the input graph. this approach effectively exploits the graph topology, and the graph structure has a strong impact on the node representation @xcite . however, encoding relations between distant nodes can be challenging by requiring more graph encoding layers, which can also propagate noise @xcite . for example, figure 1a presents a kg, for which a corresponding text is shown in figure 1b . note that there is a mismatch between how entities are connected in the graph and how their natural language descriptions are related in the text. some entities syntactically related in the text are not connected in the graph. for instance, in the sentence ''for the link prediction task, first we learn node embeddings using distmult method.'', although the entity mentions are dependent of the same verb, in the graph, the node embeddings node has no explicit connection with link prediction and distmult nodes, which are in a different connected component. this example illustrates the importance of encoding distant information in the input graph. as shown in figure 1c , a global encoder is able to learn a node representation for node embeddings which captures information from non-connected entities such as distmult. by modeling distant connections between all nodes, we allow for these missing links to be captured, as kgs are known to be highly incomplete @xcite . in contrast, the local strategy refines the node representation with richer neighborhood information, as nodes that share the same neighborhood exhibit a strong homophily: two similar entities are much more likely to be connected than at random. consequently, the local context enriches the node representation with local information from kg triples. for example, in figure 1a , gat reaches node embeddings through the gnn. this transitive relation can be captured by a local encoder, as shown in figure 1d . capturing this form of relationship also can support text generation at the sentence level. in this paper, we investigate novel graph-totext architectures that combine both global and local node aggregations, gathering the benefits from both strategies. in particular, we propose a unified graph-to-text framework based on graph attention networks (gats) @xcite . as part of this framework, we empirically compare two main architectures: a cascaded architecture that performs global node aggregation before performing local node aggregation, and a parallel architecture that performs global and local aggregations simultaneously. the cascaded architecture allows the local encoder to leverage global encoding features, and the parallel architecture allows more independent features to complement each other. to further consider fine-grained integration, we additionally consider layer-wise integration of the global and local encoders. extensive experiments show that our approaches consistently outperform recent models on two benchmarks for text generation from kgs. to the best of our knowledge, we are the first to consider integrating global and local context aggregation in graph-to-text generation, and the first to propose a unified gat structure for combining global and local node contexts. ## related work early efforts for graph-to-text generation used statistical methods @xcite @xcite . recently, several neural graph-to-text models have exhibited success by leveraging encoder mechanisms based on lstms, gnns, and transformers. ## data and preprocessing we attest the effectiveness of our models on two datasets: agenda @xcite and webnlg @xcite . agenda. in this dataset, kgs are paired with scientific abstracts extracted from proceedings of 12 top ai conferences. each instance consists of the paper title, a kg, and the paper abstract. entities correspond to scientific terms that are often multiword expressions (co-referential entities are merged). we treat each token in the title as a node, creating a unique graph with title and kg tokens as nodes. as shown in webnlg. in this dataset, each instance contains a kg extracted from dbpedia. the target text consists of sentences that verbalize the graph. we evaluate the models on the test set with seen categories. note that this dataset has a considerable number of edge relations (see in order to avoid parameter explosion, we use regularization based on the basis function decomposition to define the model relation weights @xcite . also, as an alternative, we use the levi transformation to create nodes from relational edges between entities @xcite . that is, we create a new relation node for each edge relation between two nodes. the new relation node is connected to the subject and object token entities by two binary relations, respectively. ## experiments we implemented all our models using pytorch geometric (pyg) @xcite and opennmt-py @xcite . we use the adam optimizer with β 1 = 0.9 and β 2 = 0.98. our learning rate schedule follows @xcite , with 8,000 and 16,000 warming-up steps for webnlg and agenda, respectively. the vocabulary is shared between the node and target tokens. in order to mitigate the effects of random seeds, for the test sets, we report the averages over 4 training runs along with their standard deviation. we use byte pair encoding @xcite to split entity words into smaller more frequent pieces. therefore some nodes in the graph can be sub-words. we also obtain sub-words on the target side. following previous works, we evaluate the results with bleu @xcite , meteor @xcite , and chrf++ @xcite automatic metrics and also perform a human evaluation (section 5.6). for layer-wise models, the number of encoder layers are chosen from {2, 4, 6}, and for pge and cge, the global and local layers are chosen from and {2, 4, 6} and {1, 2, 3}, respectively. the hidden encoder dimensions are chosen from {256, 384, 448} (see figure 3 ). hyperparameters are tuned on the development set of both datasets. we report the test results when the bleu score on dev set is optimal. ## conclusion in this work, we introduced a unified graph attention network structure for investigating graph-totext models that combines global and local graph encoders in order to improve text generation. an extensive evaluation of our models demonstrated that the global and local contexts are empirically complementary, and a combination can achieve state-of-the-art results on two datasets. in addition, cascaded architectures give better results compared with parallel ones. we point out some directions for future work. first, it is interesting to study different fusion strategies to assemble the global and local encodings. second, a promising direction is incorporating pre-trained contextualized word embeddings in graphs. third, as discussed in section 5.5, it is worth studying ways to diminish the gap between the reference and the generated text lengths.