XRICL : Cross-lingual Retrieval-Augmented In-Context Learning for Cross-lingual Text-to- SQL Semantic Parsing (2022)

Abstract: In-context learning using large language models has recently shown surprising results for semantic parsing tasks such as Text-to-SQL translation.Prompting GPT-3 or Codex using several examples of question-SQL pairs can produce excellent results, comparable to state-of-the-art finetuning-based models.However, existing work primarily focuses on English datasets, and it is unknown whether large language models can serve as competitive semantic parsers for other languages.To bridge this gap, our work focuses on cross-lingual Text-to-SQL semantic parsing for translating non-English utterances into SQL queries based on an English schema.We consider a zero-shot transfer learning setting with the assumption that we do not have any labeled examples in the target language (but have annotated examples in English).This work introduces the XRICL framework, which learns to retrieve relevant English exemplars for a given query to construct prompts.We also include global translation exemplars for a target language to facilitate the translation process for large language models.To systematically evaluate our model, we construct two new benchmark datasets, XSpider and XKaggle-dbqa, which include questions in Chinese, Vietnamese, Farsi, and Hindi.Our experiments show that XRICL effectively leverages large pre-trained language models to outperform existing baselines.Data and code are publicly available at https://github.com/Impavidity/XRICL.

## introduction semantic parsing is the task of translating natural language questions into meaning representations such as lambda cds @xcite , python code @xcite , and sql @xcite . more recently, text-to-sql semantic parsing has attracted attention from academia and industry due to its challenging setup and practical applications. cross-lingual text-to-sql semantic parsing @xcite @xcite aims to translate non-english utterances into sql queries based on an english schema (assuming we have an internationalized database), enabling users to query databases in non-english languages. for example, such a system could help people from around the world access the us government's open data foot_0 with natural language questions in different languages. state-of-the-art approaches for text-to-sql semantic parsing have been greatly improved by finetuning pre-trained language models as a sequenceto-sequence problem @xcite @xcite @xcite . more recently, in-context learning with large language models (llms), such as gpt-3 @xcite and codex @xcite , has emerged as a new learning paradigm. this paradigm enables effective few-shot learning without model finetuning, showing its practical and scientific value @xcite . recent papers also have shown promising results applying in-context learning to the text-to-sql task. @xcite studied if llms are already competitive text-to-sql semantic parsers without further finetuning on task-specific training data. additionally, @xcite and @xcite investigated the exemplar retrieval problem for the semantic parsing task. however, previous work mostly focused on english utterances, leaving other languages behind. it is unclear if llms are competitive for crosslingual text-to-sql with english exemplars using in-context learning. even in the mono-lingual setting (where the exemplars and the query are in the same language), many approaches are not practical beyond english due to the paucity of target language query-sql exemplars. to bridge this gap, we propose xricl, a novel framework based on llms with in-context learn- our framework introduces an additional exemplar into the llm's input context, to instruct the model to translate the target query into english and then to translate the english query into sql; this approach is inspired by recent work on chainof-thought prompting @xcite . however, in our framework, this additional exemplar is identical for all test queries, which means that we only need a single pair of translations for any english-target language pair, requiring minimal translation effort. during the inference process, the language model is expected to generate the english translation first and then the sql query. in our experiments, we find that our proposed retriever and reranker can improve the llms' cross-lingual fewshot in-context learning performance by a large margin, and further improvements can be observed by adding an additional translation exemplar. we further construct two benchmarks, xspi-der and xkaggle-dbqa, to systematically evaluate the proposed framework in many languages. for xspider, besides adopting existing work, including cspider @xcite and vspi-der @xcite , we further translate the spider dataset into farsi and hindi for evaluation. for xkaggle-dbqa, we translate the english kaggle-dbqa dataset into chinese, farsi, and hindi. experimental results show that our proposed framework improves effectiveness compared to baseline systems. our contributions are summarized as follows: ## task formulation given a database where the schema s is in english (denoted as the source language), our task is to translate a non-english (denoted the target language) example x (x includes utterance u and schema s) into a sql query a. in this work, we explore large pre-trained language models such as codex for this text-to-sql task with in-context learning. to support in-context learning, labeled candidates of (utterance, schema, sql) triples are required. since more annotated resources are available in english, we assume that the labeled candidate set d is in english. overall, in-context learning is an efficient method to leverage large pre-trained language models without expensive parameter fine-tuning. furthermore, the candidate pool can be easily expanded for better generalization to new domains. ## the xricl framework our xricl framework is shown in figure 1 , consisting of four steps: ## experimental settings in this section, we describe the datasets, implementation details, and baselines for our experiments. ## analysis 6.1 effectiveness on english text-to-sql we show that our model is comparable to other in-context learning methods for english semantic parsing. through this comparison, we show that our framework is built on a competitive backbone for text-to-sql. we use the de-retriever as the backbone model in the ablation study and compare with three recent methods, described as follows: @xcite used hard labels obtained from the generator to train the retriever. @xcite used the tree edit distance of sql queries as a similarity function: a smaller distance means better exemplar quality for the specific test instance. the ranking model is optimized to predict the target sql pair tree edit distance based on the utterance pair. @xcite designed an efficient prompt that leverages table contents for zero-shot text-to-sql. we refer the reader to the original papers for more details. ## related work in-context learning: in-context learning is a relatively new paradigm for zero-shot and few-shot learning with large-scale pre-trained language models, first proposed in gpt-3 @xcite . in-context learning for semantic parsing has been intensively investigated recently @xcite @xcite @xcite @xcite . however, most of the work considers only english, without examining the cross-lingual ability of the proposed methods. winata et al. ( 2021 ) evaluated the multilinguality of pre-trained language models on non-english multi-class classification with incontext learning. however, their task is simpler than semantic parsing tasks such as ours. to the best of our knowledge, we are the first to explore cross-lingual text-to-sql semantic parsing under the in-context learning setting. cross-lingual semantic parsing: cross-lingual semantic parsing aims to handle user utterances from multiple languages and translate them into formal representations. recent advances can be categorized into two threads: multilingual dataset creation and model development. for example, bai et al. (2018) adapted a chinese dialogue parsing dataset into english. min et al. (2019) and tuan nguyen et al. ( 2020 ) adapted the english text-to-sql dataset spider @xcite into chinese and vietnamese, which are used in this work for evaluation. some multilingual datasets with different formal representations have also been created, such as sparql @xcite and top @xcite . in terms of model development, @xcite is the most relevant to our work, which leveraged bilingual input for the semantic parsing task. however, they used rnn models and focused on multilingual representation alignment with pre-training. instead, our work focuses on representation mixup with large multilingual pretrained models. improving cross-lingual zero-shot transfer is another direction @xcite . multilingual and cross-lingual retrieval: in multilingual retrieval, the task is to retrieve relevant documents where the user queries and the corpora are in the same language. recent work takes advantage of cross-language transfer using pre-trained multilingual models @xcite @xcite . for example, @xcite used dpr to retrieve documents based on ad-hoc queries in six languages. on the other hand, cross-lingual retrievers help users find relevant documents in languages that are different from that of the queries. this task has a long history that goes back several decades @xcite , but recent work includes @xcite ; @xcite ; @xcite . for instance, @xcite created a cross-lingual open-domain question answering dataset where the system is required to retrieve passages from different languages to answer user questions. ## conclusion in this work, we proposed the xricl framework that improves in-context learning for crosslingual text-to-sql semantic parsing. the retrieveand-rerank models that we propose can learn signals from large pre-trained models (codex) to improve the quality of selected exemplars, which can further benefit the generator. by integrating prompts inspired by chain of thought, our proposed translation-p method can bridge the crosslingual gap for the generator. extensive experiments on xspider and xkaggle-dbqa demonstrate the effectiveness of our framework, which obtains state-of-the-art performance on few-shot in-context learning in most of the datasets, thus unlocking the potential of codex.