XS em PLR : Cross-Lingual Semantic Parsing in Multiple Natural Languages and Meaning Representations (2023)

Abstract: Cross-Lingual Semantic Parsing (CLSP) aims to translate queries in multiple natural languages (NLs) into meaning representations (MRs) such as SQL, lambda calculus, and logic forms. However, existing CLSP models are separately proposed and evaluated on datasets of limited tasks and applications, impeding a comprehensive and unified evaluation of CLSP on a diverse range of NLs and MRs. To this end, we present XSemPLR, a unified benchmark for cross-lingual semantic parsing featured with 22 natural languages and 8 meaning representations by examining and selecting 9 existing datasets to cover 5 tasks and 164 domains. We use XSemPLR to conduct a comprehensive benchmark study on a wide range of multilingual language models including encoder-based models (mBERT, XLM-R), encoder-decoder models (mBART, mT5), and decoder-based models (Codex, BLOOM). We design 6 experiment settings covering various lingual combinations (monolingual, multilingual, cross-lingual) and numbers of learning samples (full dataset, few-shot, and zero-shot). Our experiments show that encoder-decoder models (mT5) achieve the highest performance compared with other popular models, and multilingual training can further improve the average performance. Notably, multilingual large language models (e.g., BLOOM) are still inadequate to perform CLSP tasks. We also find that the performance gap between monolingual training and cross-lingual transfer learning is still significant for multilingual models, though it can be mitigated by cross-lingual few-shot training. Our dataset and code are available at https://github.com/psunlpgroup/XSemPLR .

## introduction cross-lingual semantic parsing (clsp) aims to translate queries in multiple natural languages (nls) into meaning representations (mrs) @xcite @xcite @xcite . as demonstrated in figure 1 , cross-lingual semantic parsing covers natural languages for geographically diverse users and various meaning representations, empowering applications such as natural language interfaces to databases, question answering over knowledge graphs, virtual assistants, smart home device control, human-robot interaction, and code generation. however, current research on clsp has three drawbacks. first, most existing research focuses on semantic parsing in english @xcite @xcite , limiting the development of multilingual information access systems for users in other languages. second, current datasets have a poor coverage of nls and mrs. although there are encouraging efforts in developing clsp models @xcite @xcite , their experiments only cover a few nls and mrs, impeding comprehensive and unified evaluation on a diverse range of tasks. third, due to the lack of a comprehensive clsp benchmark, the performance of multilingual language models on clsp is understudied. some pretrained language models are proposed to solve cross-lingual tasks such as xlm-r @xcite and mt5 @xcite , while other large language models are designed for code generation such as codex @xcite and bloom @xcite . however, little research has focused on evaluating models on clsp. in this paper, we propose xsemplr, a unified benchmark for cross-lingual semantic parsing featured with 22 natural languages and 8 meaning representations as summarized in our results show that the encoder-decoder model (mt5) yields the best performance on monolingual evaluation compared with other models. then, we pick two models with the best monolingual performance (i.e., mt5 and xlm-r) to conduct fewshot and zero-shot cross-lingual transfer learning from english to other low-resource languages. results show a significant performance gap between monolingual training (taget nl -> target nl foot_0 ) and cross-lingual transfer learning (en -> target nl). furthermore, we find that this gap can be significantly reduced by few-shot learning on target nl. we further train these two models in a multilingual setting and find such training can boost the performance in some of the languages, while, however, it usually hurts the performance in english. finally, we test two large language models codex @xcite and bloom @xcite . we find the performance gap of cross-lingual transfer learning is significant for these two models as well. our contributions are summarized as follows: (1) we propose xsemplr to unify and benchmark 9 datasets covering 5 tasks, 22 natural languages, and 8 meaning representations for cross-lingual semantic parsing; (2) we perform a holistic evaluation of 3 groups of state-of-the-art multilingual language models on xsemplr, demonstrating noticeable performance gaps of cross-lingual transfer models comparing english and other languages; we show two effective strategies for boosting performance in low-resource languages: multilingual training and cross-lingual transfer learning. ## xsemplr benchmark figure 2 shows the construction pipeline of xsem-plr. we first select 9 clsp datasets according to our design principles. then, we collect other nls of the selected datasets. finally, we clean the datasets by removing outliers and performing alignment between different languages. ## experiment setup we describe our evaluation settings and models for a comprehensive benchmark study on xsemplr. ## results and analysis ## conclusion we build xsemplr, a unified benchmark for cross-lingual semantic parsing with multiple natural languages and meaning representations. we conduct a comprehensive benchmark study on three representative types of multilingual language models. our results show that mt5 with monolingual training yields the best performance, while notably multilingual llms are still inadequate to perform cross-lingual semantic parsing tasks. moreover, the performance gap between monolingual training and cross-lingual transfer learning is still significant. these findings call for both improved semantic parsing capabilities of multilingual llms and stronger cross-lingual transfer learning techniques for semantic parsing.