Maximum B ayes S match Ensemble Distillation for AMR Parsing (2022)

Abstract: AMR parsing has experienced an unprecendented increase in performance in the last three years, due to a mixture of effects including architecture improvements and transfer learning. Self-learning techniques have also played a role in pushing performance forward. However, for most recent high performant parsers, the effect of self-learning and silver data augmentation seems to be fading. In this paper we propose to overcome this diminishing returns of silver data by combining Smatch-based ensembling techniques with ensemble distillation. In an extensive experimental setup, we push single model English parser performance to a new state-of-the-art, 85.9 (AMR2.0) and 84.3 (AMR3.0), and return to substantial gains from silver data augmentation. We also attain a new state-of-the-art for cross-lingual AMR parsing for Chinese, German, Italian and Spanish. Finally we explore the impact of the proposed technique on domain adaptation, and show that it can produce gains rivaling those of human annotated data for QALD-9 and achieve a new state-of-the-art for BioAMR.

## introduction adoption of the transformer architecture @xcite for abstract meaning representation (amr) parsing @xcite as well as pretrained language models @xcite @xcite have enabled an improvement of above 10 smatch points @xcite , the standard metric, in the last two years. data augmentation techniques have also shown great success in pushing the state-of-the-art of amr parsing forward. these include generating silver amr annotations with a trained parser @xcite , with multitask pre-training and fine-tuning @xcite as well as combining amr to source text and silver amr generation @xcite and stacked pre-training of silver data from different models -from low performance to high performance silver data @xcite . however, the latest bart-based state-of-the-art parsers, have shown diminishing returns for data augmentation. both spring @xcite and structured-bart @xcite gain a mere 0.5 smatch from self-learning, compared with over 1 point gains of the previous, less performant, models. since performance scores are already above where inter annotator agreement (iaa) is assumed to be, i.e. 83 for newswire and 79 for web text reported in @xcite , one possible explanation is that we are reaching some unavoidable performance plateau. in this work we show that we can achieve significant performance gains close to 2 smatch point with the newly proposed data augmentation technique, contrary to the results from the previous state-of-the-art systems. the main contributions of this paper are as follows: ensemble distillation @xcite integrates knowledge of different teacher models into a student model. for sequence to sequence models, e.g. machine translation, it is possible to ensemble models by combining probabilities of words given context at each time step @xcite . syntactic and semantic parsers model a distribution over graphs that is harder to integrate across teacher models in an optimal way. for particular cases like dependency parsing, it is possible to ensemble teachers based on the notion of edge attachment @xcite , which is related to the usual evaluation metric, label attachment score (las). however, amr graphs are quite complex and not explicitly aligned to words. the standard smatch @xcite ) metric approximates the np-complete problem of aligning nodes across graphs with a hill climbing algorithm. this illustrates the difficulty of achieving consensus across teachers for amr ensembling. prior work ensembling amr graphs has leveraged smatch directly or its hill climbing strategy for ensembling. the ensemble in @xcite selects, among a number of candidate amrs, the one that has the largest average smatch with respect to all sampled amrs. the ensemble in @xcite , modifies the candidate amrs to increase consensus as measured by coverage. then it selects from the union of original and modified graphs for the one with highest coverage or largest average smatch. one possible intepretation of both techniques is that of minimum bayes risk (mbr) decoding, a well established method in automatic speech recognition (asr) @xcite and machine translation (mt) @xcite . assuming that we have a model predicting a graph from an input sentence p(g | w), normal decoding entails searching among model outputs g for the one that has the highest likelihood according to the model p(g | w). mbr searches instead for the model output that minimizes the risk with respect to the distribution of possible human (gold) outputs for a given input here we suggest another bayesian interpretation, that requires less strong assumptions than mbr, a bayesian model ensemble @xcite . indeed techniques above can be seen as solving spanish sentence ŝ1 labeled english sentence e 1 gold amr g 1 machine translation spanish sentence ŝ2 alternative english sentence ê2 machine translation amr-to-text spanish sentence ŝ3 unlabeled english sentence e 3 silver amr ĝa silver amr ĝb silver amr ĝc machine translation parser-a parser-b parser-c mbse ĝ3 figure 1 : data augmentation framework: given a labeled example in english (e 1 , g 1 ), we use an amr-to-text generation system to generate an alternative input text ê2 for g 1 following @xcite . given a sentence e 3 , and various state-of-the-art off-the-shelf parser outputs (a, b, c), maximum bayes smatch ensemble (mbse) produces a single annotation for each input sentence by selecting from existing amrs or their modified versions. mbse is only applied to unlabeled english sentences to produce ĝ3 . following @xcite , we translate the english sentences to e.g. spanish, to yield new training samples (ŝ 1 , g 1 ), (ŝ 2 , g 1 ), (ŝ 3 , ĝ3 ) to train a spanish cross-lingual parser. we use the english pairs (e 1 , g 1 ), (ê 2 , g 1 ), (e 3 , ĝ3 ) to train an english parser. if we replace smatch() by an indicator function on the decoding outputs 1 @xmath0, then ĝ = arg max @xmath1. since the space of graphs is exponentially large on the input size, this would be too sparse to attain meaningful vote counts. the propagation of the uncertainty in p(m | d) through the smatch() transformation both solves the sparsity problem, and allows optimization on a space that is better related to the target metric. the method will be henceforth described here as maximum bayes smatch ensemble distillation @xcite . in what follows, we will consider three versions for ensembling, the smatch version of @xcite (graphene-smatch), the average-smatch selection of @xcite , and a greedy version of @xcite where we select the two highest smatch amrs and from that pair, keep the graph with the highest smatch with respect to the remaining graphs (greedy-select). the greedy-select algorithm is given in algorithm 1 of appendix a and performs similarly to the average-smatch of @xcite . ## silver training strategy we now describe the amr silver training strategy proposed in this work. this strategy creates high quality english and cross-lingual amr annotations for unlabeled data with mbse and alternative input sentences of gold amrs via amr-to-text. as depicted in fig. 1 , we start with 1) a set of gold-labeled (english sentence, amr) pairs, 2) a set of unlabeled english sentences and 3) pre-trained english-to-foreign language machine translation systems. assuming n off-the-shelf amr parsers, we train each of the n parsers using the gold data with their respective training procedure. more than one random seed may be trained for some parsers, leading to more than n amr parses for each input sentence. after the parsers have been trained, we use them to parse the unlabeled english text as in @xcite . interpreting the set of trained models as samples of the model distribution, we apply the mbse distillation methods described in section. 2. we apply all variations of the mbse algorithms including graphene-smatch, greedy-select and average-smatch algorithms. for english parsers, the mbse distilled amr annotations are added to the human-annotated gold treebanks for enhanced model training. for crosslingual parsers, we translate all english input sentences to the target foreign languages and train respective cross-lingual parsers with pairs of (foreign language input sentences, amr graphs in english), following @xcite . following @xcite , we also apply an amr-to-text model @xcite @xcite to generate additional sentences for human-annotated amr. we filter out the generated texts if they are too similar (@xmath2.9) or too dissimilar (@xmath3.1) to the original input texts, as measured by bleu @xcite . amr-to-text generation foot_3 is used for cross-lingual amr parser training only. @xcite and craft @xcite for bioamr (silver b ). since there were no human annotations of qald-9 corpus, we created qald-9-amr treebank. qald-9 training/test data have been annotated by 3 skilled resident human annotators with experience in amr annotations over a year. each of the annotators annotated both the train and test data sets, followed by cross validation by each other. the final annotations were adjudicated by the most experienced annotator. inter-annotator agreement (iaa) rate on a subset of 158 training sentences is over 95% in smatch. the data is made publicly available under an apache2 license. ## results to explore the effect of the proposed mbse distillation and training strategy, we consider an extensive experimental setup including standard english benchmarks (section 5.1), cross-lingual benchmarks (section 5.2) and out of domain data sets (section 5.3). 10 for model training and selection details, see appendix b and appendix c. we first provide the performance evaluation of each ensembling technique used in mbse in ## related work there have been numerous works applying ensemble/knowledge distillation @xcite to machine translation @xcite @xcite @xcite , dependency parsing @xcite and question answering @xcite @xcite . regarding ensembling amr graphs, @xcite propose choosing the amr with highest average sentence smatch to all other amrs. @xcite proposed a more complex technique capable of building new amrs by exploiting smatch's hill climbing algorithm. our work brings together ensemble distillation and smatch-based ensembling and shows that it can provide substan-tial gains over the standard self-training. @xcite show that it may be possible to use the original amr annotations devised for english as representations of equivalent sentences in other languages. @xcite ; @xcite propose annotation projection of english amr graphs to target languages to train cross-lingual parsers, using word alignments. @xcite show that one may not need alignment-based parsers for crosslingual amr, and model concept identification as a seq2seq problem. @xcite reframe semantic parsing as multilingual machine translation (mnmt) and propose a seq2seq architecture finetuned on pretrained-mbart with an mnmt objective. @xcite propose to use bilingual input to enable a model to predict more accurate amr concepts. @xcite propose a crosslingual pretraining approach via multitask learning for amr parsing. @xcite propose to use noisy knowledge distillation for multilingual amr parsing. we introduce structured-mbart and attain new sota in chinese, german, italian and spanish cross-lingual parsing by applying mbse distillation and amr-to-text. we subsume domain adaptation under data augmentation with mbse distillation, where the only difference between the two lies in the properties of the unlabeled data. the unlabeled data is drawn from the target domain for the purpose of domain adaptation rather than those similar to the source training data for data augmentation in general. ## conclusion we proposed a technique called maximum bayes smatch ensemble (mbse) distillation, which brings together smatch-based model ensembling @xcite ; @xcite and ensemble distillation @xcite of heterogeneous parsers, to significantly improve amr parsing. the technique generalizes well across various tasks and is highly effective, leading to a new single system sota in english and cross-lingual amr parsing and achieving the performance comparable to human annotated training data in domain adaptation of qald-9-amr corpus. remaining technical challenges include tokenization and alignment of an input token corresponding to more than one concept for amr parsing and identification of unknown named entities and their types for domain adaptation.