Improving AMR Parsing with Sequence-to-Sequence Pre-training (2020)

Abstract: In the literature, the research on abstract meaning representation (AMR) parsing is much restricted by the size of human-curated dataset which is critical to build an AMR parser with good performance. To alleviate such data size restriction, pre-trained models have been drawing more and more attention in AMR parsing. However, previous pre-trained models, like BERT, are implemented for general purpose which may not work as expected for the specific task of AMR parsing. In this paper, we focus on sequence-to-sequence (seq2seq) AMR parsing and propose a seq2seq pre-training approach to build pre-trained models in both single and joint way on three relevant tasks, i.e., machine translation, syntactic parsing, and AMR parsing itself. Moreover, we extend the vanilla fine-tuning method to a multi-task learning fine-tuning method that optimizes for the performance of AMR parsing while endeavors to preserve the response of pre-trained models. Extensive experimental results on two English benchmark datasets show that both the single and joint pre-trained models significantly improve the performance (e.g., from 71.5 to 80.2 on AMR 2.0), which reaches the state of the art. The result is very encouraging since we achieve this with seq2seq models rather than complex models. We make our code and model available at https://github.com/xdqkid/S2S-AMR-Parser .

## introduction abstract meaning representation (amr) parsing aims to translate a textual sentence into a directed and acyclic graph which consists of concept nodes and edges representing the semantic relations between the nodes @xcite . previous studies focus on building diverse approaches to modeling the structure in amr graphs, such as treebased approaches @xcite :arg0 (c2 / country :wiki "china" :name (n / name :op1 "china")) :arg1 (p / partner-01 :arg1 (c4 / country :wiki "germany" :name (n3 / name :op1 "germany")) :mod (i / important :degree (m / most)) :mod (t / trade-01) :location (c3 / continent :wiki "europe" in this paper, we propose to pre-train seq2seq models that aim to capture different linguistic knowledge from input sentences. to build such pre-trained models, we explore three different yet relevant seq2seq tasks, as listed in in order to investigate whether various seq2seq pre-trained models are complementary to each other in the sense that they can be learned jointly to achieve better performance, we further explore joint learning of several pre-training tasks and eval-uate its effect on amr parsing. in addition, motivated by li and hoiem (2018), we extend the vanilla fine-tuning method to optimize for both the performance of amr parsing and response preservation of the pre-trained models. detailed experimentation on two widely used english benchmarks shows that our approach substantially improves the performance, which greatly advances the state-ofthe-art. this is very encouraging since we achieve the state-of-the-art by simply making use of the generic seq2seq framework rather than designing sophisticated amr parsing models. ## baseline: amr parsing as seq2seq learning seq2seq modeling. the encoder in the transformer @xcite consists of a stack of multiple identical layers, each of which has two sub-layers: one implements the multi-head selfattention mechanism and the other is a positionwise fully connected feed-forward network. the decoder is also composed of a stack of multiple identical layers. each layer in the decoder consists of the same sub-layers as in the encoder layers plus an additional sub-layer that performs multihead attention to the output of the encoder stack. see @xcite for more details. pre-processing: linearize amr graph to target sequence. as in van noord and bos (2017), we obtain simplified amrs by removing variables and wiki links. variables in amr graphs are only necessary to indicate co-referring nodes and they do not carry any semantic information by themselves. therefore, amr graphs are first converted into amr trees by removing variables and duplicating the co-referring nodes. then newlines present in an amr tree are replaced by spaces to get a sequence. figure 1(c ) illustrates the linearization result of the amr graph in figure 1(b) . based on the data of sentences paired with linearized amr graphs, we train a seq2seq model whose outputs are also linearized amrs. post-processing: recover amr graph from target sequence. the output from transformer is an amr sequence without variables, wiki-links, and co-occurrent variables. moreover, the output may contain brackets that do not match, resulting incomplete concepts. to recover its full graph, the post-processing should restore information removed in pre-processing by assigning a unique variable to each concept, pruning duplicated and redundant material, performing wikification, and restoring co-referring nodes. meanwhile, it should fix incomplete concepts. we use the pre-processing and post-processing scripts provided by van noord and bos (2017). 1 ## seq2seq pre-training for amr parsing in this section, we first present our single pretraining approach, followed by the joint pretraining approach on two or more pre-training tasks. then we present our fine-tuning methods. ## experimentation in this section, we report the performance of our seq2seq pre-training approach to amr parsing. ## analysis and discussion in this section, we conduct more analysis on amr parsing with pre-trained models. in the following all the results are obtained on amr 2.0. ## related work we describe related work from two perspectives: pre-training and amr parsing. pre-training. pre-training a universal model and then fine-tuning the model on a downstream task have recently become a popular strategy in the field of natural language processing. previous works on pre-training can be roughly grouped into two categories. one category of approaches is to learn static word embeddings such as word2vec @xcite and glove @xcite while the other group builds dynamic pre-trained models that would also be used in downstream tasks. representative examples in the latter group in-clude @xcite , cove @xcite , elmo @xcite , openai gpt @xcite , and bert @xcite . besides the aforementioned encoder-only (e.g., bert) or decoderonly (e.g., gpt) pre-training approaches, recent studies also propose approaches to pre-training seq2seq models, such as mass @xcite , poda @xcite , pegasus @xcite , bart @xcite , and t5 @xcite . amr parsing. as a semantic parsing task that translates texts into amr graphs, amr parsing has received much attention in recent years. diverse approaches have been applied to the task. @xcite pioneer the research work on amr parsing by using a a two-stage approach: node identification followed by relation recognition. @xcite improve the first stage in the parser of @xcite by generating subgraph aligned to lexical items. to avoid conducting amr parsing from scratch, @xcite propose to obtain amr graphs from dependency trees by using a transition-based method. wang et al. (2015a) extend their previous work by introducing a new transition action to get better performance. @xcite propose a complete transition-based approach that parses sentences left-to-right in linear time. the recent neural amr parsing could be roughly grouped into two categories. on the one hand, the generic seq2seq-based approaches have been widely used for amr parsing which show competitive performance @xcite @xcite . on the other hand, to better model the graph structure on the target side, graph-based models are well studies for amr parsing which achieve the state-of-theart-performance @xcite @xcite @xcite . ## conclusion in this paper we proposed a seq2seq-based pretraining approach to improving the performance of seq2seq-based amr parsing. to this end, we designed three relevant seq2seq learning tasks, including machine translation, syntactic parsing, and amr parsing itself. then we built seq2seq pre-trained models through either single or joint pre-training tasks. detail experimentation shows that both the single and joint pre-trained models substantially improve our baseline and the performance reaches the state of the art. the accomplishment is encouraging since we achieve this simply by using the generic seq2seq framework rather than complex models.