Stacked AMR Parsing with Silver Data (2021)

Abstract: Lacking sufficient human-annotated data is one main challenge for abstract meaning representation (AMR) parsing. To alleviate this problem, previous works usually make use of silver data or pre-trained language models. In particular, one recent seq-to-seq work directly fine-tunes AMR graph sequences on the encoder-decoder pre-trained language model and achieves new state-of-the-art results, outperforming previous works by a large margin. However, it makes the decoding relatively slower. In this work, we investigate alternative approaches to achieve competitive performance at faster speeds. We propose a simplified AMR parser and a pre-training technique for the effective usage of silver data. We conduct extensive experiments on the widely used AMR2.0 dataset and the results demonstrate that our Transformer-based AMR parser achieves the best performance among the seq2graph-based models. Furthermore, with silver data, our model achieves competitive results with the SOTA model, and the speed is an order of magnitude faster. Detailed analyses are conducted to gain more insights into our proposed model and the effectiveness of the pre-training technique.

## introduction abstract meaning representation (amr) parsing aims to abstract semantics from a natural language sentence into a rooted, directed, and labeled graph, where the nodes represent concepts and edges represent semantic relations @xcite . figure 1 gives an example. one main challenge of amr parsing is the lack of large-scale annotated data, which limits the model representative ability. to alleviate the problem and boost the performance, early works propose to use silver (pseudo) data that are generated from some released amr parsing models * corresponding author. (van @xcite . apart from amr silver data, @xcite try to use other kinds of large-scale silver data to train a pre-trained model, such as constituent parsing data and machine translation data. with the development of pre-trained language models, recent works try to use pre-trained language models to enhance the model input representative ability @xcite . most of the them use pre-trained models in the model encoder side since it naturally provides powerful contextualized representations for sentences. recently, @xcite propose a seq2seq amr parser based on bart @xcite , which is one encoder-decoder fashion pre-trained language model. they first convert the amr graph into a text sequence with symbols indicating the concepts' graph positions. then, they propose to fine-tune the sentence sequence and amr graph sequence on bart, achieving large improvements compared with previous works, including those with bert. however, it makes the model relatively slower, which parses 31 tokens per second. we think there are two main reasons: 1) the 12-layer transformer decoder and 2) the longer converted graph sequences that include the added symbols. in this work, we investigate alternative approaches to achieve competitive performance at faster speeds. we propose a simplified amr parser and a pre-training technique for effective use of silver data. first, we propose a simple transformer-based seq2graph amr parser, denoted as tamr, that only needs one external bi-affine scorer @xcite for the relation classification. the remaining question is how we do concept generation and edge classification with transformer, which is usually used for encoding sequences. our answer is giving the transformer attention mechanisms more meanings. in detail, we try to demonstrate that the self-attention in the decoder captures the semantic relation that can guide establishing the connections for the concepts and the cross-attention implicitly links the concept with its surface word, which is similar to the core of attention-based machine translation. based on the inspirations, we use the copy mechanism @xcite to copy words or lemmas as candidate concepts for concept prediction, in which we treat the cross-attention between the encoder and decoder as the probability. another source of candidate concepts is the extracted concept vocabulary from the training data. for edge classification, we directly treat part of the decoder self-attention values as the edge scores between concept nodes. second, to achieve competitive performance with the current sota model, we seek to use silver data to enhance the model representative ability. specifically, we employ three different performance amr models (denoted as "father" models) to generate three different performance silver data and try to investigate several questions which are seldom discussed in previous works: 1) what are the best learning schedules to build pre-trained models with silver data and later fine-tune with the gold-standard data, respectively? 2) are all the different performance silver data beneficial for our model, even its father model lags behind our model? and 3) whether using multiple different performance silver data can provide more information than the best performance one or not, i.e., can the higher performance silver data benefits from lower performance silver data? based on the answers to these questions, which are shown in section 6.2, we propose a stack pre-training technique for effectively using silver data. we conduct extensive experiments on the commonly used amr2.0 dataset. the experimental results show that our proposed model achieves the best results among the seq2graph-based models. utilizing the silver data, our final model achieves comparable results with the current sota model, and the speed is an order of magnitude faster. pipeline-based approaches mainly consist of two steps: 1) concept identification and 2) relation identification. @xcite is the first amr parsing work (jamr) that treats concept identification as a sequence labeling problem and relation identification as a maximum-scoring connected graph searching problem, in which they also propose an influential rule-based aligner for aligning the concepts and words. @xcite treat the alignment as latent variables and propose a joint model for amr parsing. @xcite first use the attention-based copy mechanism to predict concepts in a bilstm encoder-decoder framework and then use the bi-affine scorer for edge and relation prediction based on the predicted concepts. transition-based methods aim to design a series of actions to generate the amr graph. @xcite propose to transform the sentence's dependency tree into its amr graph. @xcite ; naseem et al. (2019) use stack-lstm transition-based amr parser that transforms the sentence into the amr graph, which is different from wang et al. (2016). with the rise of transformer, astudillo et al. (2020); zhou et al. (2021) propose to use stack-transformer for transition-based amr parsing. seq2seq-based approaches convert the amr graph generation problem into a symbolic sequence generation problem, where the hierarchy structure is converted into human-defined symbols. konstas et al. (2017) propose a seq2seq-based amr parser, which uses millions of unlabeled data with self-training. van noord and bos (2017) leverage a character level seq2seq-based model and silver data, achieving promising improvements. recently, bevilacqua et al. (2021) fine-tune the the gold-standard data on bart (lewis et al., 2020), which achieves new sota performance. seq2graph-based methods generate a new concept node and its connections with previously generated concepts at one time step, thus are relatively faster than seq2seq-based methods. zhang et al. (2019b) propose a bilstm encoder-decoder-based model for several semantic tasks, including amr. cai and lam (2019) present a top-down amr parser that generates the concept nodes in a root-toleaf way. cai and lam (2020) introduce an iterative inference for the decoding process on the transformer encoder-decoder architecture. motivated by the seq2graph methods' generation process and the transformer encoder-decoder framework, we propose to adapt amr parsing into the transformer architecture. the main difference between our model and previous seq2graph models is that our model mostly relies on transformer, only added one biaffine scorer for relation classification. given one sentence @xmath0, w 2 , ..., w n , amr parsing aims to parse the sentence into an amr graph @xmath1, e}, where @xmath2, c 2 , ..., c m } is the set of concept nodes in the amr graph foot_0 and @xmath3, c j , r)|1 ≤ @xmath4, 1 ≤ @xmath5, @xmath6. r is the set of amr relations. overall, our transformer-based model consists of the following modules, i.e., input layer, encoder layer, decoder layer, concept generator, edge generator, and relation classifier. we will describe the model architecture in detail and show how to adapt the amr parsing process into transformer in the following sections. ## stack pre-training with silver data reviewing the progress of amr parsing, we can find that the performance boosting with the development of deep learning in the nlp community, especially the usage of silver data and evolution of pre-trained language models like bert. injecting bert representations @xcite into amr parsing models brings significant improvements compared with previous bertfree models @xcite , which is an effective way to potentially alleviate the data sparsity problem and enhance model representative ability. however, bert can only provide powerful representations for the sentence, which can not directly bring benefits to the decoder module in the encoder-decoder framework. recently, @xcite propose a bart-based @xcite seq2seq amr parsing model (spring), where the hierarchy structure of amr graph is represented by humandefined symbols. bart is a pre-trained seq2seq model that provides powerful representations for both encoder and decoder. thus, @xcite achieve new sota performance on amr benchmarks by simply fine-tuning bart with goldstandard amr data. we think the powerful representations of encoder and decoder contribute to the success of @xcite . however, spring runs relatively slower than previous seq2graph-based methods because of the autoregression generation process for the hierarchy symbol-based graph sequence, which needs 26 minutes to test the amr2.0 test data (about 0.88 sentences/second). unfortunately, training seq2graph models based on the seq2seq fashion pre-trained models is tricky because bart uses sub-word representations, which is difficult to establish edges for concept nodes. so, can we train a model that has the competitive performance with spring and runs as fast as the classic seq2graph models? the answer is yes and we can utilize large-scale silver data @xcite . in this work, we first employ large-scale silver data to train pre-trained tamr models for simulating the role of pre-trained language models. then we fine-tune the pre-trained model with the gold-standard amr data. to better understand the effect of silver data and investigate some valid questions that are seldom discussed before, we conduct experiments with silver data that are generated from three different performance models, i.e., jamr, tamr, and spring. figure 3 shows the overall process of our usage of these different performance silver data. first, we use the three models to parse the largescale bllip data to generate three different performance silver data. second, we use the silver data to train three different pre-trained amr models with tamr. third, we fine-tune these pre-trained models with the gold-standard amr data and investigate whether the model performance can boost or not. furthermore, we also investigate whether we can utilize all the silver data with some ensemble techniques for further improvements, which can also be seen as an ensemble of different amr parsing models. to this purpose, we propose a stack pre-training method that progressively learning with these silver data, which is depicted by the bottom workflow in figure 3 . we will discuss these details with the experimental results in section 6.2. ## analysis 6.1 effect of tamr decoder layers. the decoder plays the main role of tamr, in which the number of layers matters a lot. ## conclusion in this work, we propose a simple transformerbased amr parsing model that adapts amr parsing into the transformer architecture, in which the attention mechanisms are given more meanings of latent alignment and the connections between concepts. based on our proposed model, we conduct detailed experiments to investigate several strategies for using silver data and propose an effective stack pre-training method for ensemble different models. the experimental results show that our proposed model achieves the best performance of seq2graph-based models and demonstrate the effectiveness of using silver data. our final model achieves comparable results with the sota model, and the speed is an order of magnitude faster. detailed analyses show the effectiveness of our proposed ensemble technique of stack pre-training for using multiple silver data.