Multilingual AMR Parsing with Noisy Knowledge Distillation (2021)

Abstract: We study multilingual AMR parsing from the perspective of knowledge distillation, where the aim is to learn and improve a multilingual AMR parser by using an existing English parser as its teacher. We constrain our exploration in a strict multilingual setting: there is but one model to parse all different languages including English. We identify that noisy input and precise output are the key to successful distillation. Together with extensive pre-training, we obtain an AMR parser whose performances surpass all previously published results on four different foreign languages, including German, Spanish, Italian, and Chinese, by large margins (up to 18.8 Smatch points on Chinese and on average 11.3 Smatch points). Our parser also achieves comparable performance on English to the latest state-of-the-art English-only parser.

## introduction abstract meaning representation (amr) @xcite is a broad-coverage semantic formalism that encodes the meaning of a sentence as a rooted, directed, and labeled graph, where nodes represent concepts and edges represent relations among concepts. amr parsing is the task of translating natural language sentences into their corresponding amr graphs, which encompasses a set of natural language understanding tasks, such as named entity recognition, semantic role labeling, and coreference resolution. amr has proved to be beneficial to a wide range of applications such as text summarization @xcite , machine translation @xcite , and question answering @xcite . one most critical feature of the amr formalism is that it abstracts away from syntactic realization and surface forms. as shown in figure 1 , different the boy wants the girl to believe him. the boy's desire is for the girl to believe him. the boy wants to be believed by the girl. english sentences with the same meaning correspond to the same amr graph. furthermore, there are no explicit alignments between elements (nodes or edges) in the graph and words in the text. while this property leads to a distinct difficulty in amr parsing, it also suggests the potential of amr to work as an interlingua @xcite @xcite , which could be useful to multilingual applications of natural language understanding @xcite ). an example is given in figure 1 , we represent the semantics of semantically-equivalent sentences in other languages using the same amr graph. this defines the multilingual amr parsing problem we seek to address in this paper. multilingual amr parsing is an extremely challenging task due to several reasons. first, amr was initially designed for and heavily biased towards english, thus the parsing has to overcome some structural linguistic divergences among languages @xcite . second, the human-annotated resources for training are only available in english and none is present in other languages. moreover, since the amr graph involves rich semantic labels, the amr annotation for other languages can be laborintensive and unaffordable. third, current modeling techniques focus mostly on english. for example, existing amr aligners @xcite @xcite and widely-used pointer-generator mechanisms @xcite @xcite rely on the textual overlap between english words and amr node values (i.e., concepts). some initial attempts @xcite @xcite towards multilingual amr parsing mainly investigated the construction of pseudo parallel data via annotation projection. in this paper, we study multilingual amr parsing from the perspective of knowledge distillation @xcite @xcite , where our primary goal is to improve a multilingual amr parser by using an existing english parser as its teacher. we focus on a strict multilingual setting for developing one amr parser that can parse all different languages. in contrast to the language-specific (one parser one language) setting, our setting is more challenging yet more appealing in practice. intuitively, knowledge distillation is effective because the teacher's output provides a rich training signal for the student parser. we develop both the teacher parser and the student parser with language-agnostic seq2seq design and expect the student parser to imitate the behaviors of the teacher parser (i.e., english parser) when processing semantically-equivalent input in other languages. we first show that multilingual seq2seq pre-training, including language model and machine translation pre-training, provides an excellent starting point for model generalization across languages. we further capitalize on the idea that the student should be robust to noisy input and introduce noise by machine translation for improving student performance. to migrate the risk that the student learns the mistakes made by the teacher, the student is then fine-tuned with gold amr graphs. we present experiments on the benchmark dataset created by @xcite , covering four different languages with no training data, including german, spanish, italian, and chinese. to cover as many languages as possible, we also include the original english test set in our evaluation. on four zero-resource languages, our single universal parser consistently outperforms the previous best results by large margins (+11.3 smatch points on average and up to +18.8 smatch points). meanwhile, our parser achieves competitive results on english even compared with the latest state-ofthe-art english amr parser in the literature. to sum up, our contributions are listed below: ## experimental results the performance of amr parsing is conventionally measured by smatch score @xcite , which quantifies the maximum overlap between two amr graphs. the reported results are averaged over 3 runs with different random seeds. ## conclusion we presented a multilingual amr parser that significantly advances the state-of-the-art parsing accuracies on multiple languages. notably, the superior results are achieved with one single amr parser. our parser is trained with multiple pretraining and fine-tuning stages including a noisy knowledge distillation stage. we hope our work can facilitate the application of amr in multilingual scenarios. and comprehension. in proceedings of the 58th annual meeting of the association for computational linguistics, pages 7871-7880, online. association for computational linguistics. yaobo liang, nan duan, yeyun gong, ning wu, fenfei guo, weizhen qi, ming gong, linjun shou, daxin jiang, guihong cao, et al. 2020. xglue: a new benchmark dataset for cross-lingual pretraining, understanding and generation. arxiv preprint arxiv:2004.01401. kexin liao, logan lebanoff, and fei liu. 2018. abstract meaning representation for multi-document summarization. in proceedings of the 27th international conference on computational linguistics, pages 1178-1190, santa fe, new mexico, usa. association for computational linguistics.