GAP : A Graph-aware Language Model Framework for Knowledge Graph-to-Text Generation (2022)

Abstract: Recent improvements in KG-to-text generation are due to additional auxiliary pre-training tasks designed to give the fine-tune task a boost in performance. These tasks require extensive computational resources while only suggesting marginal improvements. Here, we demonstrate that by fusing graph-aware elements into existing pre-trained language models, we are able to outperform state-of-the-art models and close the gap imposed by additional pre-training tasks. We do so by proposing a mask structure to capture neighborhood information and a novel type encoder that adds a bias to the graph-attention weights depending on the connection type. Experiments on two KG-to-text benchmark datasets show our models are competitive while involving fewer parameters and no additional pre-training tasks. By formulating the problem as a framework, we can interchange the various proposed components and begin interpreting KG-to-text generative models based on the topological and type information found in a graph.

## introduction due to the amount of data stored in knowledge graphs (kgs) @xcite @xcite @xcite , they are important to properly transcribe into natural language sentences, making them more easily comprehensible to a larger audience. this task, termed kg-to-text, has found recent success in generating knowledge-grounded dialog responses @xcite , question answering @xcite @xcite , story generation @xcite , and event narration @xcite . kgto-text involves encoding a kg, often sparse, in figure 1 : given a graph, kg-to-text generation aims to describe the entities, relations, and its inherent structure via natural language text (grey callout). corresponding graph-text components are color-coded. recently, pre-trained language models (lms) have produced state-of-the-art results on the kgto-text generation task @xcite . these models tend to first linearize a graph into a sequence of tokens, and fine-tune on pre-trained lms such as bart @xcite , gpt @xcite , or t5 @xcite , treating the task similarly to a text-totext task. because of the performance gains caused by the self-supervised pre-training tasks, current work on kg-to-text has focused on developing pretrained tasks and large-scale unlabeled graph-text corpora, replicating the success in the text-to-text domain @xcite . however, these works particularly focus on leveraging large amounts of pre-trained data for graph-to-text specific pre-trained tasks, e.g., recovering a masked text sequence based on a given complete kg. although recent work in kg-to-text has begun to combine lms with a graph-aware approach @xcite , they do not adequately perform a graph-aware encoding, overlooking the kg's topological information. similarly, recent work in amr-to-text has begun to observe the role of graph adaptors in dense, highly parallel data, using a graph convolutional network (gcn) @xcite . instead, our framework leverages a topological attention mechanism, better adhering to the language model paradigm and giving room for interpretation. we argue and show empirically that without additional pre-trained tasks, a fully graph-aware encoding combined with the coverage of pre-trained lms such as bart @xcite , can compete with and in some cases outperform those approaches which rely on additional pre-training. by doing so, we unload the burden of requiring vast amounts of data and computational resources required for pre-training. we propose gap, a kg-to-text framework which fuses graph-aware elements into existing pre-trained lms, capturing the advantages brought forth by both model types. our framework has two main components: (i) global attention: a graph's components are first encoded using an lm to capture their global semantic information, allowing the model to utilize the lexical coverage of pre-trained lms @xcite @xcite . (ii) graph-aware attention: next, we devise a topological-aware graph attention mechanism, with entity/relation type encoding. our framework attends to and updates entity, relation, or both representations. by proposing such a framework, where graph-aware components can be interchanged, we can begin exploring explainable generative models for the kg-to-text task. we evaluate gap on two publicly available kgto-text datasets: webnlg v2.0 @xcite and eventnarrative @xcite , achieving state-of-the-art results on various natural language generation (nlg) metrics and demonstrate the value of our fully graph-aware based approach. our contributions are as follows: we make our code publically available to motivate future research. ## problem statement we aim to generate texts that describe a given kg. we define a kg to be a multi-relational graph @xmath0, e), where v is the set of entity vertices and e ⊂ v × r × v is the set of edges that connect entities with a relation from r. ## proposed framework as our model is built on top of lms such as bart, we first linearize the knowledge graph into a text string @xcite @xcite . the linearization is a sequence of all triples in the kg, interleaved with tokens that separate each triple and the triple's components @xcite . figure 2 shows an example linearization for a small knowledge graph, along with its labeled components. ## conclusion we presented gap, a graph-aware language model framework for kg-to-text generation. our framework instills the local information captured by graph attention into the global contextualized word vector representation within pre-trained lms. we demonstrated multiple configurations of our framework by introducing a graph-aware attention masking scheme and novel type encoder module, and through qualitative analysis showed that gap outperforms existing kg-to-text models, including those that rely on additional auxiliary pre-training tasks. by closely examining the different framework configurations, we introduce the capacity to interpret kg-to-text outputs through a graph's attention structure and topology.