Graph Pre-training for AMR Parsing and Generation (2022)

Abstract: Abstract meaning representation (AMR) highlights the core semantic information of text in a graph structure. Recently, pre-trained language models (PLMs) have advanced tasks of AMR parsing and AMR-to-text generation, respectively. However, PLMs are typically pre-trained on textual data, thus are sub-optimal for modeling structural knowledge. To this end, we investigate graph self-supervised training to improve the structure awareness of PLMs over AMR graphs. In particular, we introduce two graph auto-encoding strategies for graph-to-graph pre-training and four tasks to integrate text and graph information during pre-training. We further design a unified framework to bridge the gap between pre-training and fine-tuning tasks. Experiments on both AMR parsing and AMR-to-text generation show the superiority of our model. To our knowledge, we are the first to consider pre-training on semantic graphs.

## introduction abstract meaning representation (amr; @xcite ) is a semantic structure formalism. it represents the meaning of a text in a rooted directed graph, where nodes represent basic semantic units such as entities and predicates, and edges represent their semantic relations, respectively. one example is shown in figure 1(a) , with the corresponding sentence in figure 1(b) . serving as a structural representation, amr has been shown useful for nlp tasks such as text summarization @xcite @xcite , machine translation @xcite , information extraction @xcite and dialogue systems @xcite . there are two fundamental nlp tasks concerning amr, namely amr parsing @xcite @xcite @xcite @xcite and amr-totext generation @xcite al., possible :domain go-01 :polarity negative :arg0 boy (a) (b) the boy cannot go. amr parsing amr-to-text 2018; @xcite @xcite . as shown in figure 1 , the former transforms a textual input (e.g., a sentence) into a corresponding amr structure, and the latter transforms an amr input into a fluent and grammatical sentence that conveys the same meaning. a common challenge to both tasks is that amr exists in the form of a graph structure, which is difficult for neural models to learn with limited human-curated data. recently, large-scale pre-trained sequence-tosequence (seq2seq) language models @xcite have been shown useful for both tasks above. the basic idea is to linearize amr structures into a sequence form, so that both amr parsing and amr-to-text generation can be solved as standard seq2seq tasks, using a pre-trained language model fine-tuned on taskspecific data. in this way, semantic knowledge learned in self-supervised text-to-text (t2t) pretraining can benefit both text-to-graph (t2g) and graph-to-text (g2t) transformation. intuitively, structural knowledge from amr can be a useful complement to semantic knowledge from text. a natural question is whether similar self-supervision strategy can be useful for amr graphs, so that graph-to-graph (g2g) denoise autoencoder training can serve as effective addition to t2t pre-training, before a model is fine-tuned on t2g and g2t tasks. we investigate this problem in this paper. in particular, there are three questions of interest. first, as mentioned before, is g2g pre-training complementary to t2t pre-training? second, what is the most effective way to combine t2t and g2g training? third, is silver data useful for amr self-supervision training, and what is the most effective way of making use of such data? taking bart @xcite as the seqto-seq model, we introduce two strategies for g2g pre-training and propose four tasks to combine t2t and g2g training. to reduce the gap among different pre-training tasks and between pre-training and fine-tuing, we unify all pre-training tasks and fine-tuning tasks in a general framework. experimental results on standard benchmarks show that: 1) graph pre-training achieves significant improvements over the state-of-the-art systems; 2) silver data are useful for our pre-training framework; 3) our pre-training framework is a better way than finetuning to make use of silver data and; 4) our model is more robust than existing systems in unseen domains. our final models give the best reported results on both amr parsing and amr-to-text generation tasks, with a large margin of improvement over the previous best results. to our knowledge, we are the first to consider graph-to-graph selfsupervised training on semantic graphs. we release code at https://github.com/muyeby/amrbart . ## method we take bart @xcite as the basic seq2seq model (section 3.1), and introduce graph pre-training strategies (section 3.2) and an unified pre-training framework (section 3.3) for both amr parsing and amr-to-text generation. text is divided into segments and then shuffled; 5) document rotation. a document is rotated to start with a random token. in the fine-tuning, bart takes a complete text as input and maps it into a task-specific output sequence. we linearize an amr graph into a sequence, so that both amr parsing and amr-to-text generation can be performed using a seq2seq model. in addition, it allows pre-training on amr structures using bart. following @xcite , we adopt the depth-first search (dfs) algorithm, which is closely related to the linearized natural language syntactic trees @xcite . for instance, the amr graph in figure 1 is linearized into: ( <@xmath0:domain ( <@xmath1:arg0 ( <@xmath2:polarity ( <@xmath3, @xmath4, <@xmath5. to deal with such amr symbols, we follow previous work @xcite and expand the vocabulary by adding all relations and frames. in addition, to distinguish between texts and amr graphs, we add two special tokens, <@xmath6, to mark the beginning and end of amr graphs, respectively. ## experiments we evaluate the effectiveness of our model on five benchmarks and compare the results with state-ofthe-art models on amr parsing and amr-to-text generation, respectively. in addition to standard supervised training settings, we evaluate the robustness of our model in a zero-shot domain adaptation setting. ## conclusion we investigated graph pre-training as a complement to text pre-training for amr parsing and amr-totext generation tasks, using a novel unified framework with dual graph and text denoising. we find that graph pre-training is highly effective for both amr parsing and amr -to-text generation, and is a more effective way of making use of silver data compared with fine-tuning. our methods give the best results on multiple benchmarks for both tasks. framework can reduce the gap between pre-training and fine-tuning. in addition, the training curve of the proposed model converges faster while the bleu score is better than the baseline. this indicates that our model has a larger capacity than baseline.