Should Cross-Lingual AMR Parsing go Meta? An Empirical Assessment of Meta-Learning and Joint Learning AMR Parsing (2024)

Abstract: Cross-lingual AMR parsing is the task of predicting AMR graphs in a target language when training data is available only in a source language. Due to the small size of AMR training data and evaluation data, cross-lingual AMR parsing has only been explored in a small set of languages such as English, Spanish, German, Chinese, and Italian. Taking inspiration from Langedijk et al. (2022), who apply meta-learning to tackle cross-lingual syntactic parsing, we investigate the use of meta-learning for cross-lingual AMR parsing. We evaluate our models in k -shot scenarios (including 0-shot) and assess their effectiveness in Croatian, Farsi, Korean, Chinese, and French. Notably, Korean and Croatian test sets are developed as part of our work, based on the existing The Little Prince English AMR corpus, and made publicly available. We empirically study our method by comparing it to classical joint learning. Our findings suggest that while the meta-learning model performs slightly better in 0-shot evaluation for certain languages, the performance gain is minimal or absent when k is higher than 0.

## introduction abstract meaning representation @xcite represents the meaning of texts as rooted and directed acyclic graphs. amr graphs capture the underlying semantics of input texts while abstracting away from their syntactic realizations. nodes in amr graphs are not explicitly mapped to their input token. hence, it is an unanchored formalism. amrs are widely used to enhance the capabilities of nlp systems such as question answering @xcite ), text summarization @xcite , or human-robot interaction @xcite . amr was originally designed for english texts only. however, @xcite demonstrated that amr could be used for other languages such as spanish, italian, chinese, and german. since then, many approaches have adopted amr parsing for multilingual amr parsing @xcite @xcite @xcite . however, one of the main challenges for this task is the lack of data. currently, training data are only available in english @xcite and evaluation data in 6 languages: english, german, spanish, italian, chinese @xcite ), foot_0 and french @xcite . to overcome the lack of training data in target languages, previous approaches create silver training data in the target languages. this is done through machine translation @xcite under the assumption that a text conveying the same meaning should have a shared amr graph across languages. similarly, parallel corpora with english amr parsers are also employed to create silver data @xcite . another approach uses english data for training and then evaluates the model in the target language in a zero-shot manner @xcite . since evaluation data is available in five languages, most of these proposals focus on this small set of languages. in this study, our goal is to apply amr parsing for more diverse languages that have been less explored in previous work and tackle the lack of training data with k-shot learning. taking inspiration from @xcite , who applied meta-learning for k-shot cross-lingual syntactic parsing, we apply meta-learning for cross-lingual amr parsing. to examine the efficiency of the method, we compare the meta-learning approach to a classical joint learning method. our contributions to cross-lingual amr parsing are as follows: we also release the code to train and evaluate the model. foot_2 ## experimental setup silver training/validation data we aim to train a multilingual amr parser that adapts quickly to new languages, specifically french, chinese, korean, farsi, and croatian, with k examples. our method is similar to that of @xcite in applying meta-learning for a k-shot cross-lingual parsing task, but our training data is only available in english, whereas they have multilingual training data. to create multilingual training data, we apply machine translation as in previous approaches @xcite @xcite . we adopt deepl foot_5 and translate english amr training data @xcite into 13 languages: german, italian, romanian, finnish, russian, turkish, japanese, czech, dutch, polish, swedish, estonian, and indonesian. the 13 languages were chosen for compatibility with our training model, mbart @xcite , and for language diversity. they cover 5 language families: indo-european (germanic, romance, slavic), uralic, turkic, japonic, and austronesian. for each training language, there are 55,635 pairs of sentences and their corresponding amr graph. to assess the translation quality, we evaluated the training data with the referencefree evaluation metric comet @xcite . the comet score of 13 languages is 83.8±0.8. we use a total of 14 languages including english for our training data. we use spanish as the validation language and use the spanish evaluation set from amr 2.0 @xcite . for kshot evaluation during the validation and test step, k random examples from the english dev set are translated to each evaluation language. gold test data we evaluate our model in french, chinese, korean, farsi, and croatian. for french, chinese, and farsi, we employ the little prince amr corpus annotated in each language, respectively from @xcite , https://amr.isi. edu/ and @xcite . 6 for croatian and korean, we create our test sets by manually aligning the little prince corpus in each language to corresponding english amr graphs. after manual alignment, we excluded pairs exhibiting semantic discrepancies between the aligned sentence and its english counterpart, such as pairs where additional or omitted information was observed in the aligned sentences. 7 this leaves us with, respectively, 1,527 and 1,543 pairs for korean and croatian. a few examples of the final dataset are given in appendix a. 6 the original farsi dataset consists of amr concepts in farsi. since we employ amr graphs with english concepts, we use only the input texts of the corpus and graphs from the english amr corpus. 7 the first author of this article, a native korean speaker, manually aligned and filtered the data. for croatian, we automatically translated croatian text into english with google translate ( https://translate.google.com/ ) and checked the semantic discrepancy with its english counterpart. we make the test set publicly available. ## results and discussion we assessed our model across five languages in k-shot learning. surprisingly, both maml and baseline models exhibit a performance decrease when fine-tuned in 32-shot, compared to not being fine-tuned at all. we hypothesize that the mbart pre-trained model has already enough knowledge of our target languages and fine-tuning the model with only a few examples in each language may impair the model's capacity. this could also be attributed to the domain difference between the fine-tuning dataset and the test dataset. the fine-tuning dataset includes content from general fields such as online forums, journals, and web blogs, whereas the test dataset consists of the little prince, a novel written in the 1940s. consequently, the domain shift between the two datasets may have contributed to the model's inability to generalize effectively to the test domain. we provide additional analysis of our models in appendix c (effect of the number of considered languages and of the translation quality). ## related work meta-learning, also known as learning to learn, is a learning paradigm that allows a model to quickly learn a new task with only a few examples. this is made possible by the prior knowledge that the model has acquired through a series of different tasks. in cross-lingual applications, each task corresponds to a different language. the closest approach to ours is @xcite , who adopt maml for cross-lingual dependency parsing. they train a dependency parser on a set of languages using maml and then evaluate the model on unseen languages to investigate the model's ability to adapt quickly. in contrast, we focus on a semantic parsing task with an unanchored formalism. in addition, they have multilingual training data at hand, whereas we generate our silver multilingual data by machine translation from english data. another difference is that they use a graph-based biaffine model for parsing, whereas we use a seq2seq model with a linearized graph. sherborne and lapata (2023) applied meta-learning to cross-lingual sql parsing. while useful at representing (and executing) database queries expressed in natural language, sql is not a general-purpose semantic formalism like amr. to the best of our knowledge, our work is the first to apply maml for crosslingual amr parsing. ## conclusion this study investigates the effectiveness of metalearning compared to joint learning in cross-lingual amr parsing. we assess our models across lessexplored languages for amr parsing, including french, chinese, korean, farsi, and croatian. to facilitate evaluation, we develop new test sets for korean and croatian and release the data to promote amr parsing in diverse languages. our findings reveal that meta-learning exhibits minor performance gain compared to joint learning in 0-shot evaluation. the small gain diminishes for k-shot learning (when @xmath1. consequently, our results suggest that the joint learning method serves as a robust baseline, while meta-learning appears to be a sub-optimal approach for cross-lingual amr parsing. we believe that this research provides valuable insights into the comparative efficacy of metalearning and joint learning in cross-lingual amr parsing, offering important guidance for future developments in cross-lingual amr parsers.