XL - AMR : Enabling Cross-Lingual AMR Parsing with Transfer Learning Techniques (2020)

Abstract: Abstract Meaning Representation (AMR) is a popular formalism of natural language that represents the meaning of a sentence as a semantic graph. It is agnostic about how to derive meanings from strings and for this reason it lends itself well to the encoding of semantics across languages. However, cross-lingual AMR parsing is a hard task, because training data are scarce in languages other than English and the existing English AMR parsers are not directly suited to being used in a cross-lingual setting. In this work we tackle these two problems so as to enable cross-lingual AMR parsing: we explore different transfer learning techniques for producing automatic AMR annotations across languages and develop a cross-lingual AMR parser, XL-AMR. This can be trained on the produced data and does not rely on AMR aligners or source-copy mechanisms as is commonly the case in English AMR parsing. The results of XL-AMR significantly surpass those previously reported in Chinese, German, Italian and Spanish. Finally we provide a qualitative analysis which sheds light on the suitability of AMR across languages. We release XL-AMR at github.com/SapienzaNLP/xl-amr.

## introduction abstract meaning representation (amr) is a popular formalism for natural language @xcite . it represents sentences as rooted, directed and acyclic graphs in which nodes are concepts and edges are semantic relations among them. amr unifies, in a single structure, a rich set of information coming from different tasks, such as named entity recognition (ner), semantic role labeling (srl), word sense disambiguation (wsd) and coreference resolution. such representations are actively integrated in several natural language processing (nlp) applications, inter alia, information extraction @xcite , text summarization @xcite , paraphrase detection @xcite , spoken language understanding @xcite , machine translation @xcite ) and human-robot interaction @xcite . it is therefore desirable to extend amr semantic representations across languages along the lines of cross-lingual representations for grammatical annotation @xcite , concepts @xcite and semantic roles @xcite . furthermore, it could be especially useful to integrate cross-lingual semantic structures in multilingual applications of natural language understanding. a peculiar feature of the amr formalism is that it aims at abstracting away from word forms. amr graphs are unanchored, i.e., the linkage between tokens in a sentence and nodes in the corresponding graph is not explicitly annotated. hence, the feature of being agnostic about how to derive meanings from strings makes amr particularly suitable for representing semantics cross-lingually. however, amr was initially designed for encoding the meaning of english sentences. owing to this, the available resources and modelling techniques focus mostly on english, while leaving cross-lingual amr understudied. some preliminary studies showed the limits of amr as an interlingua, categorizing them as due to distinctions in the underlying ontologies or structural divergences among languages @xcite . more recent studies, instead, have provided evidence that amr or a simplified version of it can be used as a formalism for cross-lingual semantic representation, showing that it is possible to overcome some of the structural linguistic divergences @xcite . the underlying idea of this paper is that amr can be used to represent semantic information in different languages since there exist key linguistic features that are shared across languages, such as predicates, roles and conjunctions @xcite . however, developing an amr parser for multiple languages is hard because the existing annotated training resources that are sufficiently large are available in english only, and, moreover, acquiring semantic annotations for a large number of sentences is well-known to be a slow and expensive process in nlp @xcite . to this end, we aim at exploiting and developing the necessary tools and resources for enabling cross-lingual amr parsing, i.e., the task of transducing a sentence in the source language into an amr graph based on english @xcite . we present xl-amr, a cross-lingual amr parser, and study different transfer learning techniques to enable its training: i) model transfer which relies on language-independent features, ii) annotation projection relying on parallel corpora and available english amr parsers, and iii) automatic translation of the training corpora which guarantees gold amr structures. we make the following contributions: our work lies between two areas, namely, semantic parsing and cross-lingual transfer learning. semantic parsing semantic parsing is a key task required to complete the puzzle of natural language understanding @xcite , and one which is receiving growing attention in the scientific community. besides amr, various different formalisms have been proposed over the years to encode semantic structures: elementary dependency structures @xcite , prague tectogrammatical graphs @xcite , universal conceptual cognitive annotation (abend and rappoport, 2013, ucca), universal decompositional semantics (white et al., 2016, uds), inter alia. while some frameworks, such as ucca and uds, have been exploited in a cross-linguistic setting @xcite , cross-lingual amr has mainly been studied within the scope of annotation analysis works @xcite . these works point out the limitations of amr as an interlingua, and consider them partly due to the distinctions in the underlying ontologies and structural divergences among languages. @xcite also evaluate the properties of amr across languages and aim at simplifying this formalism in order to express only essential semantic features of a sentence, such as predicate roles and linguistic relations. cross-lingual amr parsing, instead, has received relatively less attention. this is largely attributable to the lack of training data and evaluation benchmarks in languages other than english. @xcite propose, to the best of our knowledge, the only cross-lingual amr parser to date and, moreover, their proposed cross-lingual amr evaluation benchmark has been released only very recently @xcite . the authors adapt a transition-based english amr parser @xcite for cross-lingual amr parsing, which is trained on silver annotated data. however, the performances it has achieved are not satisfying in terms of smatch score @xcite , mostly as a result of concept identification errors, which in turn are directly related to the usage of noisy word-to-node alignments projected from english. throughout the literature english amr parsers commonly rely on amr alignments which are automatically created using heuristics @xcite , or on pretrained aligners @xcite , treated as latent variables of the model @xcite or implicitly modelled through sourcecopy mechanisms @xcite . these alignments, however, take advantage of the fact that amr nodes and english words are highly related. 1 this dependency is therefore not suitable for crosslingual parsing since similarity between words in the sentences and concepts in the graph does not hold at large. our parser, instead, disposes of explicit and implicit amr alignments using a seq2seq model for concept identification and achieves significantly higher performance on all the tested languages. on the other hand, to account for data sparsity, xl-amr employs several common techniques in english amr parsing literature @xcite , such as anonymization and recategorization, expanding them across languages by relying on multilingual resources. transfer learning the idea behind this method is to leverage annotations available in one language, commonly english, to enable learning models that generalize to languages where labelled resources are scarce @xcite . different techniques include annotation projection, machine translation and language-independent feature-based models. extensive works in this direction exist, applied to different nlp tasks, i.e., wsd @xcite , srl @xcite , dependency parsing @xcite , concept representation (conia and navigli, 2020), etc. in cross-lingual amr parsing, annotation projection is employed by @xcite , who produce cross-lingual silver amr annotations by exploiting parallel sentences selected from the europarl corpus @xcite : english sentences are parsed using an english parser @xcite and the resulting graphs are associated with the corresponding parallel sentences. however, the data on which amreager was trained is very different from those used to produce the silver annotations, thus affecting the quality and reliability of the amr graphs produced. here we test two different techniques: we conduct experiments with annotation projection using europarl for comparison, and, in addition, we use translation techniques to produce better quality training corpora. this leads to significant improvements and provides evidence that better quality data -and models -allow for using amr as an interlingua. ## cross-lingual amr in what follows we first formalize the task (section 3.1) and then detail our cross-lingual amr parser (section 3.2) and our proposed silver data creation methods (section 3.3). finally, we list the preand postprocessing cross-lingual techniques and resources we employ (section 3.4). la città di tel aviv dista meno di 650 miglia dal territorio iraniano. la ciudad de tel aviv está a menos de 1.046 km del territorio iraní. die stadt tel aviv ist weniger als 650 meilen vom iranischen territorium entfernt. ## experiments we now present a set of experiments for crosslingual amr parsing when using different training techniques and the silver data we created (see section 3.3). we discuss the results of our multiple settings and compare with previous approaches performing cross-lingual amr parsing. test bed we evaluate on the abstract meaning representation 2.0 -four translations (damonte and cohen, 2020), a corpus containing translations of the test split of 1371 sentences from the ldc2017t10 (amr 2.0), in chinese (zh), german (de), italian (it) and spanish (es). this data is designed for use in cross-lingual amr parsing (available to all ldc subscribers). dataset in section 3.3, we explained the two projection approaches for obtaining cross-lingual amr data, i.e., parsents-silveramr and goldamr-silvertrns. for the first approach, inspired by @xcite , and for comparison purposes, we 3 stanza does not provide a ner model for italian. for the second approach, instead, i.e., goldamr-silvertrns, we choose amr 2.0 as gold dataset and translate the sentences into chinese, german, italian and spanish. for german, italian and spanish, for both translating and back-translating the sentences we use the machine translation models made available by tiedemann and thottingal (2020, opus-mt). 5 for chinese, instead, since opus-mt does not provide translation models, we employ the released mass foot_6 @xcite supervised neural translation models. then, to filter out less accurate translations, we compute the cosine similarity between dense semantic representations of the original english sentence and its back-translated counterpart. to embed the sentences we use laser @xcite , a state-ofthe-art model for sentence embeddings. details on the number of instances per language and for each silver data approach are shown in training configurations we conduct experiments following different training approaches: systems we denote the variations of xl-amr, based on the above training configurations, as xl-amr data where, @xmath0, trans, amr}, par referring to the data produced with parsents-silveramr approach, trans to goldamr-silvertrns approach, amr to the amr 2.0 english gold standard, and data+ refers to combining par or trans with amr. the one of damonte and cohen (2018, amreager multilingual), henceforth amreager. we compare the results of the xl-amr variants with the projection method of amreager on the gold dataset, i.e., amr 2.0 -four translations. we remark that we do not consider the results of their machine translation 7 method, since, as emphasised by the authors, it is not informative in terms of cross-lingual properties of amr @xcite because it performs english amr parsing. we provide details of our model hyperparameters in appendix c. ## qualitative analysis we manually check the predictions of xl-amr in order to establish the nature of the mistakes based on the smatch score between the gold and predicted amr graphs and determine their severity. then, we observe how xl-amr handles the translation divergences, i.e., linguistic distinctions that make transfer across languages difficult @xcite . ## conclusion we explored transfer learning techniques to enable high performance cross-lingual amr parsing. we created silver data based on annotation projection through parallel sentences and machine translation, on which we trained xl-amr, a cross-lingual amr parser that achieves the highest results reported to date on chinese, german, italian and spanish. a qualitative evaluation showed that xl-amr is able to handle most of the structural divergences among languages. the performance of xl-amr together with the qualitative analysis suggests that carefully modeling cross-lingual amr parsing leads to the production of suitable amr structures across languages. it would therefore be promising to extend this line of our research to exploit larger multilingual semantic resources, in order to further improve the parsing quality. these amr representations could then be integrated into downstream crosslingual tasks to investigate their added value.