Bootstrapping Multilingual AMR with Contextual Word Alignments (2021)

Abstract: We develop high performance multilingual Abstract Meaning Representation (AMR) systems by projecting English AMR annotations to other languages with weak supervision. We achieve this goal by bootstrapping transformer-based multilingual word embeddings, in particular those from cross-lingual RoBERTa (XLM-R large). We develop a novel technique for foreign-text-to-English AMR alignment, using the contextual word alignment between English and foreign language tokens. This word alignment is weakly supervised and relies on the contextualized XLM-R word embeddings. We achieve a highly competitive performance that surpasses the best published results for German, Italian, Spanish and Chinese.

## introduction abstract meaning representation graphs are rooted, labeled, directed, acyclic graphs representing sentence-level semantics @xcite . in the example shown in figure 1 , the sentence the boy wants to go is parsed into an amr graph. the nodes of the amr graph represent the amr concepts, which may include normalized surface symbols e.g. boy, propbank frames @xcite e.g. want-01, go-02 as well as other amr-specific constructs. edges in an amr graph represent the relations between concepts. in this example :arg0, :arg1 correspond to standard roles of propbank. one distinctive aspect of amr annotation is the lack of explicit alignments between nodes in the graph and words in the sentences. since such alignments are essential for training many of presentday amr parsers, there have been various efforts to link the amr concepts to their corresponding span of words @xcite et @xcite @xcite . a significant emphasis of this paper is on deriving these alignments for multilingual amr parsers. even though by nature amr is biased towards english, recent work has evaluated the potential of amr to work as an interlingua. @xcite and @xcite categorize and propose refinements for divergences in the annotation between english and chinese as well as czech amrs. anchiêta and pardo (2018) import the corresponding amr annotation for each sentence from the english annotated corpus and revisit the annotation to adapt it to portuguese. however, @xcite show that it may be possible to use the original amr annotations devised for english as representation for equivalent sentences in other languages without any modification despite the translation divergence. this defines the problem of multilingual amr parsing that we seek to address in this paper -given a sentence in a foreign language, recover the amr graph originally designed for its english translation. we implement multilingual amr parsers for german, spanish, italian and chinese. in this paper we propose that transformer-based multilingual word embeddings can be a useful tool for addressing the problem of multilingual amr parsing. besides using contextual word embeddings as input token embeddings, we leverage them for annotation projection, where existing amr annotations for english are projected to a target language by using contextual word alignments. in our experiments, we employ xlm-roberta large @xcite as the multilingual pre-trained transformer model. we show that our proposed procedure achieves competitive results as some of the classical methods for text-to-amr alignment. furthermore, such a procedure is easily scalable to the 100 languages that xlm-r is trained on. we also combine different techniques for concept alignments and amr parser training which significantly improve performance over the base models. for concept alignment, we combine the proposed contextual word alignments with previously established alignment techniques utilizing matching rules tailored to amr as well as machine translation aligners @xcite . for amr parser training, we pre-train an amr parser on the treebanks of different languages simultaneously and subsequently finetune on each language. this is analogous to the techniques used for silver data pre-training @xcite in amr parsing and multi-lingual pre-training @xcite in machine translation. finally, we conduct a detailed error analysis of the multilingual amr parsing. one of the major errors we have found involves synonymous concepts, which share the same meaning as the original concepts in english, but differ in spellings. while this error is mainly caused by the fact that the multilingual word embeddings bridge non-english input tokens to english concepts, it also highlights the highly lexical nature of smatch scoring @xcite which does not take synonymous concepts into consideration. we also elaborate upon error analysis of the direct comparison between our proposed annotation projection method using contextual word alignment and a previous baseline, using fast align. the rest of the paper is organized as follows: in section 2, we discuss related work. in section 3, we present our main proposal on annotation projec-tion based on contextual word alignments. in section 4, we describe various combination approaches that improve the multilingual parser performances significantly. these include combining word-toconcept alignments, using multi-lingual treebanks and combining human-annotated and synthetic treebanks. in section 5, we discuss experimental results. in sections 6 and 7, we present detailed error analyses. we conclude the paper in section 8. ## related work multilingual amr. there have been significant advances in amr parsing for languages other than english. previous studies @xcite @xcite investigated amr annotations for a variety of different languages such as chinese, czech, spanish and brazilian portuguese. @xcite automatically parse the logical representation for sentences in spanish, italian, german and japanese, which is then converted to amr using a small set of rules. while much of this work, along with studies such as @xcite ; @xcite , produces amr graphs whose nodes were labeled with words from the target language, damonte and cohen (2018) developed amr parsers for english and used parallel corpora for annotation projection to train italian, spanish, german, and chinese parsers that recover the amr graph originally designed for the english translation. their main results showed that the new parsers can overcome certain structural differences between languages. similar to @xcite , we also train multilingual amr parsers by projecting english amr annotation to target foreign languages (german, spanish, italian and chinese), but we depart from their approach in the specifics of the annotation projection by exploring contextual word alignments directly derived from multilingual contextualized word embeddings. while both procedures utilize parallel corpora, the annotation projection of @xcite requires additional supervised training of their statistical word aligner. our proposed contextualized word alignment is however unsupervised in nature. alternatively, a recent study by @xcite showed that one may in fact not need alignmentbased parsers for cross-lingual amr, rather modelling concept identification as a seq2seq problem. in this paper, we will compare our results to both @xcite and @xcite . word vector alignment techniques. traditional word alignment methods often use parallel corpora and ibm alignment models @xcite as well as improved versions @xcite . more recently, there have been an advent of techniques that align vector representation of words from varying levels of supervision @xcite . often word vectors are learned independently for each language and then a mapping from source language vectors to target language vectors with a bilingual dictionary is developed @xcite @xcite . to reduce the need for bilingual supervision, the iterative method of starting from a minimal seed dictionary and alternating with learning the linear map was employed by a recent body of work @xcite @xcite . the work most similar to ours is @xcite where the authors obtain contextual embedding alignments from multilingual bert @xcite and subsequently improve the alignments via finetuning using supervised parallel corpora. our contextual word alignment between two parallel sentences may be thought of as an adaptation of their contextual word retrieval task. however, we refrain from any finetuning of the contextual embeddings and show that the contextual word alignments from the off-the-shelf xlm-r model achieves results competitive to the word alignments by fast-align (see @xcite ). this suggests the potential for inexpensive, massive scaling of amr parsing up to 100 languages on which xlm-r is trained. ## annotation projection we adopt a transition-based parsing approach for amr parsing following @xcite @xcite . these produce an amr graph g from an input sentence s by predicting instead an action sequence a from s as a sequence to sequence problem. this action sequence applied to a state machine m produces then the desired target graph as @xmath0, s). transition-based parsers require the action sequence for each graph in the training data. this is determined by a rule-based oracle @xmath1, s) which relies on external word-to-node alignments. in all the subsequent experiments we will use the oracle and action set from (fernandez astudillo et al., 2020). ## combination approaches we apply three types of combination techniques to the multilingual amr parsers, trained by projecting english annotations using contextual word alignments derived from the multilingual contextual word embeddings, each of which improves the parser performance significantly. ## error analysis we carried out an error analysis of 56 german sentences parsed by the best performing model trained on the combination of amr2.0 and squad training data. statistics of the various errors are depicted in figure 4 . top 5 most frequent errors include (i) introduction of synonymous concepts, (ii) missing concepts, (iii) incorrect roles, (iv) target tokens in amr concepts, (v) incorrect parsing of multi-sentence as an instance of conjunction. ## word alignment error analysis we compared the annotation projection for amr1.0 between fast align and the contextual alignment. as noted in given an english token, the contextualized alignment necessarily aligns it to a foreign language word. furthermore, since embeddings are contextual and pre-trained with large amounts of data, they are robust to non frequent alignment pairs. the difference between contextualized alignment and fast align for their coverage is most noticeable for compounds. a german counterpart of english non -tariff is nichttarifäre. while contextualized alignment aligns nichttarifäre to non, which is subsequently aligned to the concept "-" for polarity, fast align leaves nichttarifäre unaligned. such difference is evidenced in the parser performance on negations realized in diverse morphologies. comparing the amr1.0 parser performance on negations between fast align (baseline iii in ## conclusion and future directions in this paper we propose to use transformer-based multilingual word embeddings for annotation projection of amr annotations. we show that our proposed procedure achieves competitive results as some of the classical methods for text-to-amr alignment. we apply combination techniques to concept alignments and amr parser training, which significantly improve performance over the base models. we also provide a detailed error analysis of the multilingual amr parsing. given pre-trained transformer-based multilingual word embeddings, contextual word alignment proves to be a useful avenue for overcoming differences amongst languages and addressing the multilingual amr problem with weak supervision. moreover, our annotation projection procedure not only achieves a highly competitive performance for german, spanish, italian and chinese but also permits zero-shot learning to other languages included in the training set of the underlying xlm-r multilingual transformer. future work may include diversifying input texts using amr2text @xcite generation which can address the difference in results between machine translated and human translated test data. the potential of the amr parser to overcome translation divergence also points to its utility in an end-to-end multilingual translation system, bypassing the need for supervised parallel corpora for machine translation system training.