XLPT - AMR : Cross-Lingual Pre-Training via Multi-Task Learning for Zero-Shot AMR Parsing and Text Generation (2021)

Abstract: Due to the scarcity of annotated data, Abstract Meaning Representation (AMR) research is relatively limited and challenging for languages other than English. Upon the availability of English AMR dataset and English-to- X parallel datasets, in this paper we propose a novel cross-lingual pre-training approach via multi-task learning (MTL) for both zeroshot AMR parsing and AMR-to-text generation. Specifically, we consider three types of relevant tasks, including AMR parsing, AMR-to-text generation, and machine translation. We hope that knowledge gained while learning for English AMR parsing and text generation can be transferred to the counterparts of other languages. With properly pretrained models, we explore four different finetuning methods, i.e., vanilla fine-tuning with a single task, one-for-all MTL fine-tuning, targeted MTL fine-tuning, and teacher-studentbased MTL fine-tuning. Experimental results on AMR parsing and text generation of multiple non-English languages demonstrate that our approach significantly outperforms a strong baseline of pre-training approach, and greatly advances the state of the art. In detail, on LDC2020T07 we have achieved 70.45%, 71.76%, and 70.80% in Smatch F1 for AMR parsing of German, Spanish, and Italian, respectively, while for AMR-to-text generation of the languages, we have obtained 25.69, 31.36, and 28.42 in BLEU respectively. We make our code available on github https://github.com/xdqkid/XLPT-AMR .

## introduction abstract meaning representation (amr) @xcite is a widely used formalism that represents the semantics of a sentence with a directed and acyclic graph. figure 1 (b) shows an example amr graph where the nodes such as * corresponding author: junhui li. restricted by the availability of annotated corpora, most of previous studies on amr focus on english while very few studies are for chinese and portuguese @xcite @xcite . cross-lingual amr research, however, has received relatively less attention. in fact, crosslingual amr has mainly been studied in the scope of annotation works @xcite . till recently, @xcite demonstrate that amr annotated for english can be used as cross-lingual semantic representations, and propose to conduct cross-lingual amr parsing via annotation projection and machine translation. @xcite follow the same line and create large-scale silver data to boost the performance of cross-lingual amr parsing. @xcite focus on multilingual amr-to-text generation for twenty one different languages. the aforementioned studies consider amr parsing and amr-to-text generation separately. in this paper, we formalize both amr parsing and amr-to-text generation as sequence-tosequence (seq2seq) learning and propose a novel and effective approach to cross-lingual amr, which is illustrated in figure 1 . upon the availability of the english amr dataset and english-to-x parallel datasets (@xmath0, spanish, italian} in this paper), our purpose is to boost the performance of zero-shot amr parsing and text generation in (a) parallel sentences english the doctors gave her medication and it's made her much better. ## related work we describe related studies on amr from three perspectives: english amr parsing, english amrto-text generation, and cross-lingual amr. english amr parsing. amr parsing is a task that translates a sentence into a directed and acyclic graph @xcite . according to the approaches to modeling the structure in amr graphs, previous studies on amr parsing for english can be broadly grouped into several categories, which are tree-based approaches @xcite , graph-based approaches @xcite @xcite , transition-based approaches @xcite @xcite @xcite , sequence-to-sequence (seq2seq) approaches @xcite @xcite @xcite , and sequence-to-graph (seq2graph) approaches @xcite @xcite . english amr-to-text generation. as an inverse task of amr parsing, amr-to-text generation aims to write a sentence from an amr graph. early studies on this task rely on grammar-based approaches @xcite . more recent studies propose to regard amr-totext generation as a machine translation or seq2seq task @xcite @xcite . however, seq2seq approaches tend to lose structural information in amr graphs since they simply linearize amr graphs into sequences before feeding them into the models. to prevent information loss caused by linearization, a variety of graph-tosequence approaches have been proposed to better model structural information @xcite @xcite @xcite @xcite @xcite @xcite . by taking advantages of strong pre-trained language models, recent studies achieve new state of the art @xcite @xcite . ## cross-lingual pre-training in this section, we first present the background of our pre-training approach (section 3.1), followed by the description of cross-lingual pre-training tasks (section 3.2). then we present our joint pre-training (section 3.3). for simplicity, in the following we use german as a representative to describe our approach to german amr parsing and amr-to-text generation. ## fine-tuning methods to fine-tune a pre-trained model, we create a fine-tuning dataset from english annotated amrs (i.e.,amr2.0). given english-amr parallel data f en , f am r , we use an english-to-german translator to translate the english sentences into german sentences, thus obtain trilingual parallel dataset @xmath1, f de , f am r . as our goal is to improve the performance of zero-shot amr parsing and amr-to-text generation, our primary fine-tuning tasks are german amr parsing and amr-to-text generation. moreover, we could include the other four fine-tuning tasks as auxiliary tasks when necessary, i.e., english amr parsing and amr-to-text generation, as well as english-to-german and german-to-english translation. once the fine-tuning dataset is ready, we can finetune a pre-trained model with different methods. the vanilla fine-tuning method that fine-tunes a pretrained model on the dataset of a primary task is a natural choice. we can also fine-tune a pre-trained model jointly over all fine-tuning tasks, or over the primary tasks plus specifically chosen fine-tuning tasks that are relevant. in the following we explore and compare four different fine-tuning methods. ## experimentation in this section, we report the performance of our approach to amr parsing and amr-to-text generation for non-english languages, including german (de), spanish (es), and italian (it). the models are pre-trained and fine-tuned on english data and one of either de, es, or it, and are evaluated in the target language. ## conclusions in this paper we proposed a cross-lingual pretraining approach via multi-task learning for zeroshot amr parsing and amr-to-text generation. upon english amr dataset and english-to-x parallel datasets, we pre-trained models on three types of relevant tasks, including amr parsing, amrto-text generation, and machine translation. we also explored and compared four different finetuning methods. experimentation on the multilingual amr dataset shows that our approach greatly advances the state of the art.