Modeling Graph Structure via Relative Position for Text Generation from Knowledge Graphs (2021)

Abstract: We present Graformer, a novel Transformer-based encoder-decoder architecture for graph-to-text generation. With our novel graph self-attention, the encoding of a node relies on all nodes in the input graph - not only direct neighbors - facilitating the detection of global patterns. We represent the relation between two nodes as the length of the shortest path between them. Graformer learns to weight these node-node relations differently for different attention heads, thus virtually learning differently connected views of the input graph. We evaluate Graformer on two popular graph-to-text generation benchmarks, AGENDA and WebNLG, where it achieves strong performance while using many fewer parameters than other approaches.

## introduction a knowledge graph (kg) is a flexible data structure commonly used to store both general world knowledge @xcite and specialized information, e.g., in biomedicine @xcite and computer vision @xcite . generating a natural language description of such a graph (@xmath0. it is therefore important for kg-based question answering @xcite , datato-document generation @xcite and interpretability of kgs in general @xcite . recent approaches to @xmath1: the encoder first computes vector representations of the graph's nodes, the decoder then uses them to predict the text sequence. typical encoder choices are graph neural networks based on message passing between direct neighbors in the graph @xcite or variants of transformer @xcite that apply self-attention on all nodes together, including those that are not directly connected. to avoid losing information, the latter approaches use edge or node labels from the shortest path when computing the attention between two nodes @xcite . assuming the existence of a path between any two nodes is particularly problematic for kgs: a set of kg facts often does not form a connected graph. we propose a flexible alternative that neither needs such an assumption nor uses label information to model graph structure: a transformerbased encoder that interprets the lengths of shortest paths in a graph as relative position information and thus, by means of multi-head attention, dynamically learns different structural views of the input graph with differently weighted connection patterns. we call this new architecture graformer. following previous work, we evaluate graformer on two benchmarks: (i) the agenda dataset @xcite , i.e., the generation of scientific abstracts from automatically extracted entities and relations specific to scientific text, and (ii) the webnlg challenge dataset @xcite , i.e., the task of generating text from dbpedia subgraphs. on both datasets, graformer achieves more than 96% of the state-of-the-art performance while using only about half as many parameters. in summary, our contributions are as follows: (1) we develop graformer, a novel graph-to-text architecture that interprets shortest path lengths as relative position information in a graph self-attention network. (2) graformer achieves competitive performance on two popular kg-to-text generation benchmarks, showing that our architecture can learn about graph structure without any guidance other than its text generation objective. (3) to further investigate what graformer learns about graph structure, we visualize the differently connected graph views it has learned and indeed find different attention heads for more local and more global graph information. interestingly, direct neighbors are considered particularly important even without any structural bias, such as introduced by a graph neural network. (4) analyzing the performance w.r.t. different input graph properties, we find evidence that graformer's more elaborate global view on the graph is an advantage when it is important to distinguish between distant but connected nodes and truly unreachable ones. ## related work most recent approaches to graph-to-text generation employ a graph neural network (gnn) based on message passing through the input graph's topology as the encoder in their encoder-decoder architectures @xcite @xcite . as one layer of these encoders only considers immediate neighbors, a large number of stacked layers can be necessary to learn about distant nodes, which in turn also increases the risk of propagating noise @xcite . other approaches @xcite base their encoder on the transformer architecture @xcite and thus, in each layer, compute self-attention on all nodes, not only direct neighbors, facilitating the information flow between distant nodes. like graformer, these approaches incorporate information about the graph topology with some variant of relative position embeddings @xcite . they, however, assume that there is always a path between any pair of nodes, i.e., there are no unreachable nodes or disconnected subgraphs. thus they use an lstm @xcite to compute a relation embedding from the labels along this path. however, in contrast to the amr 2 graphs used for their evaluation, kgs are frequently disconnected. graformer is more flexible and makes no assumption about connectivity. furthermore, its relative position embeddings only depend on the lengths of shortest paths i.e., purely structural information, not labels. it thus effectively learns differently connected views of its input graph. deficiencies in modeling long-range dependencies in gnns have been considered a serious limitation before. various solutions orthogonal to our approach have been proposed in recent work: by 2 abstract meaning representation incorporating a connectivity score into their graph attention network, @xcite manage to increase the attention span to k-hop neighborhoods but, finally, only experiment with @xmath2. our graph encoder efficiently handles dependencies between much more distant nodes. @xcite define an additional neighborhood based on euclidean distance in a continuous node embedding space. similar to our work, a node can thus receive information from distant nodes, given their embeddings are close enough. however, @xcite compute these embeddings only once before training whereas in our approach node similarity is based on the learned representation in each encoder layer. this allows graformer to dynamically change node interaction patterns during training. recently, ribeiro et al. (2020) use two gnn encoders -one using the original topology and one with a fully connected version of the graph -and combine their output in various ways for graph-totext generation. this approach can only see two extreme versions of the graph: direct neighbors and full connection. our approach is more flexible and dynamically learns a different structural view per attention head. it is also more parameter-efficient as our multi-view encoder does not need a separate set of parameters for each view. ## the graformer model graformer follows the general multi-layer encoderdecoder pattern known from the original transformer @xcite . in the following, we first describe our formalization of the kg input and then how it is processed by graformer. ## learned graph structure we visualize the learned attention bias γ for different relative graph positions r ij (cf. § 3.3; esp. eq. ( 7 )) after training on agenda and webnlg in fig. 3 . the eight attention heads (x-axis) have learned different weights for each graph position r ij (y-axis). note that agenda has more possible r ij values because n δ = 6 whereas we set n δ = 4 for webnlg. for both datasets, we notice that one attention head primarily focuses on global information (5 for agenda, 4 for webnlg). agenda even dedicates head 6 entirely to unreachable nodes, showing the importance of such nodes for this dataset. in contrast, most webnlg heads suppress information from unreachable nodes. for both datasets, we also observe that nearer nodes generally receive a high weight (focus on local information): in fig. 3b , e.g., head 2 concentrates solely on direct incoming edges and head 0 on direct outgoing ones. graformer can learn empirically based on its task where direct neighbors are most important and where they are not, showing that the strong bias from graph neural networks is not necessary to learn about graph structure. ## conclusion we presented graformer, a novel encoder-decoder architecture for graph-to-text generation based on transformer. the graformer encoder uses a novel type of self-attention for graphs based on shortest path lengths between nodes, allowing it to detect global patterns by automatically learning appropriate weights for higher-order neighborhoods. in our experiments on two popular benchmarks for text generation from knowledge graphs, graformer achieved competitive results while using many fewer parameters than alternative models.