Structural Adapters in Pretrained Language Models for AMR -to- T ext Generation (2021)

Abstract: Pretrained language models (PLM) have recently advanced graph-to-text generation, where the input graph is linearized into a sequence and fed into the PLM to obtain its representation. However, efficiently encoding the graph structure in PLMs is challenging because such models were pretrained on natural language, and modeling structured data may lead to catastrophic forgetting of distributional knowledge. In this paper, we propose StructAdapt, an adapter method to encode graph structure into PLMs. Contrary to prior work, StructAdapt effectively models interactions among the nodes based on the graph connectivity, only training graph structure-aware adapter parameters. In this way, we incorporate task-specific knowledge while maintaining the topological structure of the graph. We empirically show the benefits of explicitly encoding graph structure into PLMs using StructAdapt, outperforming the state of the art on two AMR-to-text datasets, training only 5.1% of the PLM parameters.

## introduction data-to-text tasks aim to generate meaningful and coherent natural language text that faithfully conveys structured data. some examples of structured information include tables @xcite , knowledge graphs (kgs) @xcite and abstract meaning representation (amr) @xcite . in this work, we focus on amr-to-text generation where the goal is to generate a fluent and grammatical sentence that is faithful to a given amr graph (see figure 1a ). amr is a semantic formalism that has received much research interest @xcite @xcite @xcite and has been shown to benefit downstream tasks 1 our code and checkpoints are available at https://github.com/ukplab/structadapt . such as text summarization @xcite and machine translation @xcite . both statistical @xcite and neural methods @xcite have been investigated for amrto-text generation, and dominant methods make use of graph neural networks (gnns) @xcite or transformers @xcite for representing the input graph. pretrained language models (plms) @xcite @xcite have been shown useful as a general text representation method, giving much improved results on a wide range of tasks @xcite . however, they cannot be directly leveraged to benefit amr-to-text generation, and more generally graph-to-text generation, due to the structural nature of the input. one solution is to transform the structured input into a se-quence, which can be directly fed into plms (see figure 1b ). recent studies @xcite @xcite transform amrs into sequences by top-down linearization @xcite . it has been shown that such linearized graph representation can be used to fine-tune a plm and improve graph-to-text generation performances @xcite . the above methods, however, suffer from two salient limitations. first, linearized graph structures are different in nature from natural language. as a result, knowledge from large-scale pretraining intuitively cannot be fully transferred, and finetuning a sentence representation using linearized graphs can lead to catastrophic forgetting of such distributional knowledge @xcite . second, a linearized representation weakens structural information in the original graphs by diluting the explicit connectivity information (i.e., which nodes are connected to each other), and plms must infer how edge connections are specified in the sequence. this fact was also observed by @xcite , @xcite and @xcite , who show that gnn encoders outperform sequential encoders for amr-to-text generation without pretraining. to mitigate the issues, we aim to explicitly encode the graph data into a plm without contaminating its original distributional knowledge. to this end, we propose structadapt, a novel structureaware adapter that effectively allows leveraging the input graph structure into plms (see figure 1c ). the main idea is to add layer-wise modules, which extract information from the pretrained layers and make use of it in a graph-structure encoding. as shown in figure 2 , structadapt employs a graph convolution in order to learn representations built upon the graph connectivity over the plm encoder. because structadapt is added to each encoder layer, deep integration of linguistic knowledge and graph knowledge can be achieved. during finetuning, only the adapter parameters are trained, whereas the plm parameters remain unchanged, in contrast to previous methods based on the graph linearizations that fine-tune all model parameters. empirically we show that structadapt significantly outperforms linearized fine-tuning baselines and naive sequential adapters @xcite . moreover, structadapt is more robust to different graph linearizations, better treats reentrancies (nodes with more than one entering edge) and long-range node dependencies. our proposed models, based on structadapt, surpass the current state of the art on ldc2017t10 and ldc2020t02 datasets by up to 3.1 bleu points, training only 5.1% of the original plm parameters. ## related work fine-tuning for graph-to-text generation. while previous approaches @xcite @xcite @xcite have shown that explicitly encoding the graph structure is beneficial, fine-tuning plms on linearized structured data has established a new level of performance in data-to-text generation @xcite @xcite . our work can be seen as integrating the advantage of both graph structure encoding and plms, using a novel adapter module. @xcite employ cycle consistency to improve the adequacy of generated texts from amrs using gpt-2 @xcite , whereas @xcite train a classifier to rank candidate generations based on the semantic fidelity. ribeiro et al. (2020a) investigate encoder-decoder plms for graph-to-text generation, and show that task-specific pretraining can lead to notable improvements and that plms benefit much more from the graph structure of amrs than of kgs. @xcite explore the extent to which plms are invariant to graph linearization, finding that models trained on canonical linearizations fail to generalize to meaning-preserving alternatives. compared to this line of work, which tunes all plm parameters, our method obtains a further 19x reduction in task-specific parameters, tuning only 5.1% of the parameters while achieving state-of-the-art performance, being more robust to permutations of the graph representation and better encoding larger graphs. lightweight fine-tuning. recently, different approaches have emerged as an alternative training strategy in order to avoid fine-tuning all parameters of a plm. @xcite train a lightweight "side" network that is fused with the pretrained model via summation. li and liang (2021) propose to prepend a trainable continuous prefix as an alternative to adapters, maintaining comparable performance in data-to-text tasks using fewer trained parameters. @xcite develop a method to automatically search prompts in the continuous space and evaluate it in few-shot nlu tasks. @xcite propose adversarial reprogramming attempts to learn task-specific word embeddings to customize the language model for the downstream task. adapter-based approaches @xcite @xcite @xcite introduce a small number of task specific parameters, keeping the underlying pretrained model fixed. @xcite propose an adapter method to arbitrary tasks and languages by learning modular language and task representations. the above works are related to structadapt as it trains much fewer parameters, but also different because they do not explicitly encode the input structure, whereas structadapt directly aims to encode it. ## structural adapter we propose structadapt, a lightweight alternative to injecting structural inductive bias foot_0 into plms. we first describe the intuition in §4.1 and define our method formally in §4.3. ## experiments our models are initialized with pre-trained t5 @xcite , but our approach can be combined with other plms such as bart @xcite . our implementation is based on hugging face transformer models @xcite . we use t5 base for all experiments and report results with t5 large for the test sets. 6 we use the adam optimizer (kingma and ba, 2015) and employ a linearly decreasing learning rate schedule without warm-up. bleu is used for the stopping criterion. following recent work @xcite , we evaluate our proposed models on ldc2017t10 and ldc2020t02 corpora. evaluation. we evaluate the results with bleu @xcite and chrf++ @xcite metrics. we also report the meaning (m) component of the mf-score @xcite , which measures how well the source amr graph can be reconstructed from the generated sentence. we use bertscore @xcite allowing a semantic evaluation that depends less on the surface forms. finally, we also perform a human evaluation ( §5.2). ## graph representation evaluation in this section, we explore how different graph properties impact the models' abilities to encode the input graph structure. ## conclusion we presented structadapt, a novel adapter architecture to explicitly model graph structures into pretrained language models, providing an extensive evaluation of our approach and showing that it achieves state-of-the-art results on two amr-totext benchmarks, training much fewer parameters. we also found that structadapt is more effective when encoding complex graphs, when trained on fewer datapoints, and is more robust to different graph linearizations and reentrancies. in future work, we plan to consider other graph-to-text tasks, such as those based on knowledge graphs.