Bootstrapping Multilingual Semantic Parsers using Large Language Models (2023)

Abstract: Despite cross-lingual generalization demonstrated by pre-trained multilingual models, the translate-train paradigm of transferring English datasets across multiple languages remains to be a key mechanism for training task-specific multilingual models. However, for many low-resource languages, the availability of a reliable translation service entails significant amounts of costly human-annotated translation pairs. Further, translation services may continue to be brittle due to domain mismatch between task-specific input text and general-purpose text used for training translation models. For multilingual semantic parsing, we demonstrate the effectiveness and flexibility offered by large language models (LLMs) for translating English datasets into several languages via few-shot prompting. Through extensive comparisons on two public datasets, MTOP and MASSIVE, spanning 50 languages and several domains, we show that our method of translating data using LLMs outperforms a strong translate-train baseline on 41 out of 50 languages. We study the key design choices that enable more effective multilingual data translation via prompted LLMs.

## introduction enabling language technologies across several languages is an important goal for serving a diverse range of users in an inclusive manner. recent advances in large-scale self-supervised multilingual language models hold immense promise in bridging the quality gap that currently exists between english and many other low resource languages @xcite @xcite . even though multilingual models exhibit cross-lingual generalization, getting meaningful performance across several languages still requires significant amounts of task-specific labeled data. we consider the problem of automatically synthesizing semantic parsing datasets across several * work done during an internship at google research languages. semantic parsing @xcite @xcite is the task of mapping natural language text into an executable logical-form. for example, given a user instruction (x) : "wake me up by 5 am", mapping it to the logical-form (y): [in:create_alarm [sl:date_time 5 am ]]. manual annotation of queries with their logical forms requires human expertise which makes data collection across multiple languages challenging. a common approach to automatic multilingual dataset creation is translating existing english datasets into target languages. prior methods utilize an off-the-shelf machine translation model for translating the english utterance into the target language x @xmath0, followed by projecting language specific components in the english logicalform y eng to obtain the logical-form y tgt in the target language @xcite @xcite @xcite . the projection step is often learned independent of the translation service, resulting in poor generalization across languages. in this work we aim to utilize the few-shot generalization abilities exhibited by large language models (llms) @xcite @xcite for bootstrapping semantic parsing datasets across fifty languages. we propose a recipe of using llms to translate an english semantic parsing dataset containing (utterance, logical-form) pairs: d @xmath1, y i eng )} into a corresponding dataset in a target language: d @xmath2, y i tgt )}. the generated dataset d tgt is then used to train a semantic parser in the target language. our method uses a small amount of manually translated semantic parsing examples to teach the llm how to translate english examples in the target language via in-context learning @xcite . figure 1 describes our data-translation pipeline which we refer to as llm-t ( § 3). in contrast figure 1 : proposed semantic parsing data translation pipeline using llms ( § 3): with the help of human translators, we first collect translations of a small seed set of english examples in the target language (e.g. hindi; § 3.1). given a new english example, a small subset from this initial seed set of examples with their respective translations is chosen to prompt the llm ( § 3.2). the prompted llm translates the given english example in the target language. we repeat this process for each example in the english training data to generate a training dataset in the target language. to ensure high-quality of the resulting dataset, we generate diverse translations via top-p (nucleus) sampling ( § 3.3) and apply consistency filtering ( § 3.4). to prior translation based methods that involved a two-staged process requiring different modules, our method uses the llm to jointly translate an english (x eng , y eng ) pair directly into the target language (x tgt , y tgt ). we identify two important choices that make the llm translated data more effective for training a downstream parser: we perform experiments on two multilingual semantic parsing datasets: mtop @xcite and @xcite . on 4 out of 5 languages in mtop and 41 out of 50 languages in massive, our method llm-t outperforms taf @xcite , a strong baseline that utilizes a supervised translation service ( § 5.1). further, we see that llm-t achieves 93% of the performance obtained by "fully-supervised" models that use 30× more manually translated examples ( § 5.2). we justify the importance of generat-ing multiple translations using sampling, filtering out inconsistent examples, and using larger-sized llms in improving translated data quality ( § 5.3). finally, we perform an error analysis of our parser and show the key sources of disagreements between the model predictions and the ground truth ( § 5.4). ## background in this section, we provide an overview of semantic parsing and prior translation-based methods for creating multilingual semantic parsing datasets. ## our method: prompting llms for dataset translation our goal is to learn a multilingual semantic parser capable of parsing user queries in many languages. towards this goal, we propose a method for generating multilingual training datasets via few-shot prompting of an llm to translate existing english datasets into several languages. in contrast to prior approaches, we jointly perform example translation by prompting an llm with a few exemplars of translating english (x eng , y eng ) pairs to target language (x tgt , y tgt ) pairs. figure 1 describes our data-translation method which we refer to as llm-t. with the help of human translators we first collect a small seed set of exemplar translations used for prompting the llm ( § 3.1). given an input english example, we dynamically construct the llm prompt by identifying a relevant subset of seed exemplars ( § 3.2). the llm translates the english example into the target language by in-context learning from the exemplars provided in the prompt. instead of decoding the most likely translation, we generate multiple diverse translations ( § 3.3) using top-p (nucleus) sampling @xcite . while sampling improves the text diversity, it can lead to more noisy generations. we filter out the noisy generations using a simple string-match based technique before training a parser on the translated data ( § 3.4). ## experimental set-up we describe our experimental setup in this section. ## results and analysis we first present downstream performance of semantic parsing models trained on data generated by our method ( § 5.1) and compare with zero-shot setting, few-shot setting, and the taf method @xcite . we then compare our method against the "full-shot" skyline where we utilize the original training datasets that were manually translated with the help of human annotators in the target languages ( § 5.2). we then present an analysis of different design choices that result in effective data translation using llm-t ( § 5.3). finally, we present an error analysis to show the key sources of disagreements between the parser predictions and the ground truth ( § 5.4). all the experiments use our corrected em metric ( § 4; evaluation metric). ## related work multilingual semantic parsing multilingual semantic parsers are typically initialized with a foundation model @xcite pre-trained on vast amounts of multilingual data @xcite @xcite followed by supervised training on synthetic or real multilingual datasets. a standard approach for constructing multilingual datasets is to translate and localize english datasets with the help of multilingual speakers or machine translation. for example, mtop @xcite , @xcite , and multiatis++ @xcite were constructed by translating top @xcite , slurp @xcite , and atis @xcite ) respectively through human translators. machine translation based methods machine translation based approaches continue to be important for multilingual task-specific models @xcite @xcite @xcite including semantic parsing. machine translation can either be used during the inference time to translate a user query into english for feeding it to an english-only model. this approach is referred to as translate-test @xcite . a more common way of using machine translation is in the form of data augmen-tation, referred as translate-train where english text in training data is translated into several languages @xcite @xcite @xcite @xcite . in practice, translate-train methods tend to outperform translate-test methods while also reducing the latency associated with translating text during the inference time @xcite . ## conclusion we present a method of utilizing large language models (llms) for bootstrapping multilingual semantic parsers across several languages. in comparison to using off-the-shelf translation services that rely on significant amounts of human supervision, we demonstrate that prompting self-supervised llms can be a more effective and scalable alter-native for dataset translation. we find that generating multiple diverse translations using sampling techniques followed by consistency-based filtering make the translated datasets more effective for training multilingual semantic parsers. on 41 out of 50 typologically diverse languages within two large datasets spanning several domains, our method outperforms a strong translate-train method that utilizes a supervised translation service.