Pushing the Limits of AMR Parsing with Self-Learning (2020)

Abstract: Abstract Meaning Representation (AMR) parsing has experienced a notable growth in performance in the last two years, due both to the impact of transfer learning and the development of novel architectures specific to AMR. At the same time, self-learning techniques have helped push the performance boundaries of other natural language processing applications, such as machine translation or question answering. In this paper, we explore different ways in which trained models can be applied to improve AMR parsing performance, including generation of synthetic text and AMR annotations as well as refinement of actions oracle. We show that, without any additional human annotations, these techniques improve an already performant parser and achieve state-of-the-art results on AMR 1.0 and AMR 2.0.

## introduction abstract meaning representation (amr) are broad-coverage sentence-level semantic representations expressing who does what to whom. nodes in an amr graph correspond to concepts such as entities or predicates and are not always directly related to words. edges in amr represent relations between concepts such as subject/object. amr has experienced unprecedented performance improvements in the last two years, partly due to the rise of pre-trained transformer models @xcite @xcite , but also due to amr-specific architecture improvements. a non-exhaustive list includes latent node-word alignments through learned permutations @xcite , minimum risk training via reinforce @xcite , a sequence-to-graph modeling of linearized trees with copy mechanisms and re-entrance features @xcite and more recently a highly performant graph-sequence iterative refinement model @xcite and a hard-attention transition-based parser (f. @xcite , both based on the transformer architecture. given the strong improvements in architectures for amr, it becomes interesting to explore alternative avenues to push performance even further. amr annotations are relatively expensive to produce and thus typical corpora have on the order of tens of thousands of sentences. in this work we explore the use self-learning techniques as a means to escape this limitation. we explore the use of a trained parser to iteratively refine a rule-based amr oracle @xcite to yield better action sequences. we also exploit the fact that a single amr graph maps to multiple sentences in combination with amr-to-text @xcite , to generate additional training samples without using external data. finally we revisit silver data training @xcite . these techniques reach 77.3 and 80.7 smatch @xcite on amr1.0 and amr2.0 respectively using only gold data as well as 78.2 and 81.3 with silver data. ## baseline parser and setup to test the proposed ideas, we used the amr setup and parser from (f. @xcite with improved embedding representations. this is a transitionbased parsing approach, following the original amr oracle in @xcite and further improvements in @xcite . briefly, rather than predicting a graph g from a sentence s directly, transition-based parsers predict instead an action sequence a. this action sequence, when applied to a state machine, produces the graph @xmath0, s). this turns the problem of predicting the graph into a sequence to sequence problem, but introduces the need for an oracle to determine the action sequence @xmath1, s). as in previous works, the oracle in (f. a. et al., 2020) is rule-based, relying on external word-to-node alignments @xcite to determine action sequences. it however force-aligns unaligned nodes to suitable words, notably improving oracle performance. as parser, (f. a. et al., 2020) introduces the stack-transformer model. this is a modification of the sequence to sequence transformer @xcite to account for the parser state. it modifies the cross-attention mechanism dedicating two heads to attend the stack and buffer of the state machine m (a, s). this parser is highly performant achieving the best results for a transition-based parser as of date and second overall for amr2.0 and tied with the best for amr1.0. the stack-transformer is trained as a conventional sequence to sequence model of p(a | s) with a cross entropy loss. we used the full stack and full buffer setting from (f. @xcite with same hyper-parameters for training and testing with the exception of the embeddings strategy detailed below. all models use checkpoint averaging @xcite of the best 3 checkpoints and use a beam size of 10 foot_0 while decoding. we refer to the original paper for exact details. unlike in the original work, we use robertalarge, instead of roberta-base embeddings, and we feed the average of all layers as input to the stack-transformer. this considerably strengthens the baseline model from the original 76.3/79.5 for the amr1.0/amr2.0 development sets to 77.6/80.8 smatch foot_1 . this baseline will be henceforth referred to as (f. a. et al., 2020) plus strong embeddings (+se). ## oracle self-training as explained in section 2, transition-based parsers require an oracle @xmath2, s) to determine the action sequence producing the graph @xmath3, s). previous amr oracles @xcite @xcite are rule based and rely on external wordto-node alignments. rule-based oracles for amr are sub-optimal and they do not always recover the original graph. the oracle score for amr 2.0, measured in smatch, is 98.1 (f. a. et al., 2020) and 93.7 for @xcite . in this work, we explore the idea of using a previously trained parser, p(a | s) to improve upon an existing oracle, initially rule-based. for each training sentence s with graph g and current oracle action sequence a * , we first sample an action sequence ã ∼ p(a | s). both ã and a * are run through the state machine m () to get graphs g and g * respectively. we then replace a this procedure is guaranteed to either increase smatch, shorten action length or leave it unaltered. the downside is that many samples have to be drawn in order to obtain a single new best action, we therefore refer to this method as mining. starting from the improved (f. a. et al., 2020), we performed 2 rounds of mining, stopping after less than 20 action sequences were obtained in a single epoch, which takes around 10 epochs foot_2 . between rounds we trained a new model from scratch with the new oracle to improve mining. this led to 2.0% actions with better smatch and 3.7% shorter length for amr1.0 and 2.8% and 3.2% respectively for amr2.0. this results in an improvement in oracle smatch from 98.0 to 98.2 for amr 1.0 and 98.1 to 98.3 for amr 2.0. ## self-training with synthetic text amr abstracts away from the surface forms i.e. one amr graph corresponds to many different valid sentences. the amr training data, however, provides only one sentence per graph with minor exceptions. amr 1.0 and amr 2.0 training corpora have also only 10k and 36k sentences, respectively, making generalization difficult. we hypothesize that if the parser is exposed to allowable variations of text corresponding to each gold graph, it will learn to generalize better. to this end, we utilize the recent state-of-theart amr-to-text system of @xcite , a generative model based on fine-tuning of gpt-2 @xcite . we use the trained model p(s | g) to produce sentences from gold amr graphs. for each graph g in the training data, we generate 20 sentences via sampling s ∼ p(s | g) and one using the greedy best output. we then use the following cycle-consistency criterion to filter this data. we use the improved stack-transformer parser in two separate gpt-2-based amr-to-text systems were fine-tuned using amr 1.0 and amr 2.0 train sets and then sampled to generate the respective text data foot_3 and conventional training was carried out over the extended dataset. as shown in ## self-training with synthetic amr a trained parser can be used to parse unlabeled data and produce synthetic amr graphs, henceforth synamr. although these graphs do not have the quality of human-annotated amrs, they have been shown to improve amr parsing performance @xcite . the performance of prior works is however not any more comparable to current systems and it is therefore interesting to revisit this approach. for this, we used the improved (f. a. et al., 2020) parser of sec. 2 to parse unlabeled sentences from the context portion of squad-2.0, comprising 85k sentences and 2.3m tokens, creating an initial synamr corpus. this set is optionally filtered to reduce the training corpus size for amr 2.0 experiments and is left unfiltered for amr 1.0, due to its smaller size. the filtering combines two criteria. first, it is easy to detect when the transition-based system produces disconnected amr graphs. outputs with disconnected graphs are therefore filtered out. second, we use a cycle-consistency criteria as in section 4 whereby synthetic text is generated for each synthetic amr with @xcite . for each pair of original text and generated text, the synamr is filtered out if bleu score @xcite is lower than a pre-specified threshold, 5 in our experiments. because the amrto-text generation system is trained on the humanannotated amr only, generation performance may be worse on synthetic amr and out of domain data. consequently we apply bleu-based filtering only to the input texts with no out of vocabulary (oov) tokens with respect to the original humanannotated corpus. after filtering, the synamr data is reduced to 58k sentences. following prior work, we tested pre-training on synamr only, as in @xcite , or on the mix of human-annotated @xcite and then fine-tuned on the amr1.0 or amr2.0 corpora. ## conclusions in this work foot_5 , we explored different ways in which trained models can be applied to improve amr parsing performance via self-learning. despite the recent strong improvements in performance through novel architectures, we show that the proposed techniques improve performance further, achieving new state-of-the-art on amr 1.0 and amr 2.0 tasks without the need for extra human annotations.