Investigating Pretrained Language Models for Graph-to-Text Generation (2021)

Abstract: Graph-to-text generation aims to generate fluent texts from graph-based data. In this paper, we investigate two recent pretrained language models (PLMs) and analyze the impact of different task-adaptive pretraining strategies for PLMs in graph-to-text generation. We present a study across three graph domains: meaning representations, Wikipedia knowledge graphs (KGs) and scientific KGs. We show that approaches based on PLMs BART and T5 achieve new state-of-the-art results and that task-adaptive pretraining strategies improve their performance even further. We report new state-of-the-art BLEU scores of 49.72 on AMR-LDC2017T10, 59.70 on WebNLG, and 25.66 on AGENDA datasets - a relative improvement of 31.8%, 4.5%, and 42.4%, respectively, with our models generating significantly more fluent texts than human references. In an extensive analysis, we identify possible reasons for the PLMs’ success on graph-to-text tasks. Our findings suggest that the PLMs benefit from similar facts seen during pretraining or fine-tuning, such that they perform well even when the input graph is reduced to a simple bag of node and edge labels.

## introduction graphs are important data structures in nlp as they represent complex relations within a set of objects. for example, semantic and syntactic structures of sentences can be represented using different graph representations (e.g., amrs, @xcite ; semantic-role labeling, @xcite and knowledge graphs (kgs) are used to describe factual knowledge in the form of relations between entities @xcite @xcite . graph-to-text generation, a subtask of data-totext generation @xcite , aims to create fluent natural language text to describe an input graph (see figure 1 ). this task is important for nlp applications such as dialogue generation @xcite and question answering @xcite . recently, it has been shown that structured meaning representation, such as amr or kg, can store the internal state of a dialog system, providing core semantic knowledge @xcite or can be the result of a database query for conversational qa @xcite . moreover, dialog states can be represented as kgs to encode compositionality and can be shared across different domains, slot types and dialog participators @xcite . transfer learning has become ubiquitous in nlp and pretrained transformer-based architectures @xcite have considerably outperformed prior state of the art in various downstream tasks @xcite @xcite . in this paper, we analyze the applicability of two recent text-to-text pretrained language models (plms), bart @xcite and t5 @xcite , for graph-to-text generation. we choose these models because of their encoderdecoder architecture, which makes them particularly suitable for conditional text generation. our study comprises three graph domains (meaning representations, wikipedia kgs, and scientific kgs). we further introduce task-adaptive graph-to-text pretraining approaches for plms and demonstrate that such strategies improve the state of the art by a substantial margin. while recent works have shown the benefit of explicitly encoding the graph structure in graph-totext generation @xcite @xcite @xcite , to name a few), our approaches based on plms consistently outperform these models, even though plms -as sequence models -do not exhibit any graph-specific structural bias. 2 simply representing the graph as a linear traversal (see figure 1 ) leads to remarkable generation performance in the presence of a strong language model. in our analysis we investigate to what extent fine-tuned plms make use of the graph structure represented in the graph linearization. we notably observe that plms achieve high performance on two popular kg-totext benchmarks even when the kg is reduced to a mere bag of node and edge labels. our contributions are the following: graph-to-text learning. various neural models have been proposed to generate sentences from graphs from different domains. @xcite propose the first neural approach for amrto-text generation that uses a linearized input graph. prior approaches for kg-to-text generation train text-to-text neural models using sequences of kg triples as input @xcite @xcite . recent approaches @xcite @xcite @xcite @xcite propose architectures based on gnns to directly encode the graph structure, whereas other efforts @xcite @xcite ) inject the graph structure information into transformer-based architectures. the success of those approaches suggests that imposing a strong relational inductive bias into the graph-to-text model can assist the generation. pretrained language models. pretrained transformer-based models, such as bert @xcite , xlnet @xcite , or roberta @xcite , have established a qualitatively new level of baseline performance for many widely used natural language understanding (nlu) benchmarks. generative pretrained transformer-based methods, such as gpt-2 @xcite , bart @xcite , and t5 @xcite , are employed in many natural language generation (nlg) tasks. @xcite were the first to employ gpt-2, a decoder-only plm, for amr-to-text generation and use cycle consistency to improve the adequacy. in contrast, we are the first to investigate bart and t5 models, which have both a transformer-based encoder and decoder, in amrto-text generation. recently, @xcite and @xcite demonstrate state-of-the-art results in different data-to-text datasets, employing gpt-2 and t5 models respectively. radev et al. (2020) propose dart, a new data-to-text dataset, and train a bart model gradually augmenting the webnlg training data with dart data. @xcite explore scaffolding objectives in plms and show gains in low-resource graph-to-text settings. different from the above works, we focus on a general transfer learning strategies for graph-to-text generation, investigating task-adaptive pretraining approaches, employing additional collected task-specific data for different plms (bart and t5) and benchmarks. in addition, we provide a detailed analysis aimed at explaining the good performance of plms on kgto-text tasks. recently, @xcite explored taskadaptive pretraining strategies for text classification. while our lma (see §3) is related to their dapt as both use a self-supervised objective on a domainspecific corpus, they notably differ in that dapt operates on the model input while lma models the output. we are the first to show the benefits of additional task-specific pretraining in plms for graph-to-text tasks. ## datasets we evaluate the text-to-text plms on three graph-to-text benchmarks: amr (ldc2017t10), webnlg @xcite , and agenda @xcite . we chose those datasets because they comprise different domains and are widely used in prior work. amr. abstract meaning representation (amr) is a semantic formalism that represents the meaning of a sentence as a rooted directed graph expressing "who is doing what to whom" @xcite . in an amr graph, nodes represent concepts and edges represent semantic relations. an instance in ldc2017t10 consists of a sentence annotated with its corresponding amr graph. following @xcite , we linearize the amr graphs using the penman notation (see figure 1a ). 4 webnlg. each instance of webnlg contains a kg from dbpedia @xcite and a target text with one or multiple sentences that describe the graph. the test set is divided into two partitions: seen, which contains only dbpedia categories present in the training set, and unseen, which covers categories never seen during training. their union is called all. following previous work @xcite , we prepend h , r , and t tokens before the head entity, the relation and tail entity of a triple (see figure 1b ). agenda. in this dataset, kgs are paired with scientific abstracts extracted from proceedings of ai conferences. each sample contains the paper title, a kg, and the corresponding abstract. the kg contains entities corresponding to scientific terms and the edges represent relations between these entities. this dataset has loose alignments between the graph and the corresponding text as the graphs were automatically generated. the input for the models is a text containing the title, a sequence of all kg entities, and the triples. the target text is the paper abstract. we add special tokens into the triples in the same way as for webnlg. ## experiments we modify the bart and t5 implementations released by hugging face @xcite in order to adapt them to graph-to-text generation. for the kg datasets, we add the h , r , and t tokens to the models' vocabulary. we add all edge labels seen in the training set to the vocabulary of the following previous works, we evaluate the results with bleu @xcite , me-teor @xcite , and chrf++ @xcite metrics. we also use mover-score @xcite , bertscore @xcite , and bleurt @xcite ) metrics, as they employ contextual and semantic knowledge and thus depend less on the surface symbols. additionally, we perform a human evaluation (cf. §5.4) quantifying the fluency, semantic adequacy and meaning similarity of the generated texts. ## influence of the graph structure we conduct further experiments to examine how much the plms consider the graph structure. to this end, we remove parentheses in amrs and replace h , r , and t tokens with neutral separator tokens, denoted •, for kgs, such that the graph structure is only defined by the order of node and edge labels. if we shuffle such a sequence, the graph structure is thus completely obscured and the input effectively becomes a bag of node and edge labels. see figure 2 for an example of both a correctly ordered and a shuffled triple sequence. ## conclusion we investigated two pretrained language models (plms) for graph-to-text generation and show that the pretraining strategies, language model adaptation (lma) and supervised task adaptation (sta), can lead to notable improvements. our approaches outperform the state of the art by a substantial margin on three graph-to-text benchmarks. moreover, in a human evaluation our generated texts are perceived significantly more fluent than human references. examining the influence of the graph structure on the text generation process, we find that plms may not always follow the graph structure and instead use memorized facts to guide the generation. a promising direction for future work is to explore ways of injecting a stronger graphstructural bias into plms, thus possibly leveraging their strong language modeling capabilities and keeping the output faithful to the input graph.