QA meleon: Multilingual QA with Only 5 Examples (2023)

Abstract: The availability of large, high-quality datasets has been a major driver of recent progress in question answering (QA). Such annotated datasets, however, are difficult and costly to collect, and rarely exist in languages other than English, rendering QA technology inaccessible to underrepresented languages. An alternative to building large monolingual training datasets is to leverage pre-trained language models (PLMs) under a few-shot learning setting. Our approach, QAmeleon, uses a PLM to automatically generate multilingual data upon which QA models are fine-tuned, thus avoiding costly annotation. Prompt tuning the PLM with only five examples per language delivers accuracy superior to translation-based baselines; it bridges nearly 60% of the gap between an English-only baseline and a fully-supervised upper bound fine-tuned on almost 50,000 hand-labeled examples; and consistently leads to improvements compared to directly fine-tuning a QA model on labeled examples in low resource settings. Experiments on the TyDiqa-GoldP and MLQA benchmarks show that few-shot prompt tuning for data synthesis scales across languages and is a viable alternative to large-scale annotation.1

## introduction question answering (qa) has seen impressive progress in recent years enabled by the use of large pre-trained language models @xcite @xcite , and the availability of high-quality benchmarks @xcite ; * equal contribution. see contributions section for details. 1 we release the multilingual qa synthetic data used for fine-tuning them at https://github.com/google -research-datasets/qameleon . @xcite . many qa datasets frame the task as reading comprehension, where the question is about a paragraph or document and the answer is a span therein. advances in qa modeling have been primarily reported for english, which offers a considerable amount of highquality training data compared to other languages. more recently, efforts have focused on the creation of multilingual qa benchmarks such as tydi qa (10 languages; @xcite , mlqa (6 languages; @xcite , and xquad (10 languages; @xcite . among these, only tydi qa is genuinely large-scale, mlqa and xquad are limited to an evaluation set due to the high cost and labor required to collect data across languages. as a result, efforts to localize qa models to new languages have been primarily focusing on zero-shot approaches. recent proposals include using machine translation to approximate training data for supervised learning @xcite , and data augmentation via generating synthetic questions for new languages @xcite . both approaches rely on transfer from english, which leads to a dependence on translation artifacts @xcite and a bias towards the linguistic characteristics of english, which is not the best source for all target languages @xcite . however, annotating a minimally sized data sample can potentially overcome these limitations while incurring significantly reduced costs compared to full dataset translation @xcite . in this paper, we argue that a few-shot approach in combination with synthetic data generation and existing high-quality english resources can mitigate some of the above-mentioned artifacts. beyond question answering, multilingual approaches right: model performance on the multilingual qa task. we report average exact match (em) across all languages on the tydiqa-goldp dataset @xcite . we synthesize these directions into qameleon, an approach for bootstrapping multilingual qa systems, with as few as five examples in a new target language (see figure 1 ). we use gold annotations to prompt-tune a plm in order to automatically generate multilingual qa data, which is then used to fine-tune a qa model. we find that qameleon delivers accuracy superior to zero-shot methods and competitive translation-based baselines, and in some cases competes with the fully supervised upper bound. 2 experiments on the tydi qa @xcite and mlqa @xcite benchmarks show that few-shot prompt tuning @xcite scales across languages, significantly outperforms prompt engineering @xcite with the same number of labeled examples, and is a viable alternative to large-scale annotation. our contributions include (a) a new approach to bootstrapping a multilingual qa system; qameleon prompt-tunes a plm with as few as five gold examples to automatically generate multilingual qa data which is then used to fine-tune a qa model; (b) a series of experimental results showing significant improvements over existing approaches in the few-shot regime, ranging from 12% absolute accuracy on tydiqa-goldp @xcite over an english-only baseline and 4% absolute accuracy over a competitive translatetrain baseline; (c) extensive analysis of the behavior of qameleon in zero shot and low resource regimes, on different multilingual qa datasets, and in comparison to prompt-engineering. throughout this work, we will assume the availability of d en , a large qa dataset in english (source language). this assumption corresponds to the observation that most large-scale qa datasets @xcite @xcite contain examples exclusively in english. for languages other than english, we assume that only small datasets d l,n are available for training (e.g., @xmath0. we will also assume that sets u l of unlabeled passages are available for all target languages. our task will be to synthesize qa data in each target language l in order to fine-tune qa models on l directly. in the rest of this section we formally describe three ways of synthesizing qa data and give further details on the two scenarios we consider, ''english-only'' and ''few-shot''. ## experimental setup we evaluate the synthetic data generation approaches presented in section 2 across various languages on two benchmark datasets, which we discuss below. we also describe various model configurations, and comparison systems before presenting our results. ## data analysis these were in the range of 47,000-53,000 qa examples for tydiqa-goldp, and 89,000 for mlqa. the varying size of the data across languages is due to the filtering described in section 3. in some languages (e.g., telugu) generation is more noisy leading to fewer data points. we conjecture this is due to the plm being exposed to less data representing these languages during pre-training. we further hypothesize that a more multilingual pre-training of plms could potentially lead to better quality data across all languages. machine translation (mt) creates the same number of data points as the source training set. for tydiqa-goldp, the english training contains 3,696 data points ( figure 3 shows the distribution of various question types for individual languages and on average across all languages. for each language, synthetically generated questions were first translated to english and then grouped into categories (inner circle) based on their first word and sub-categories (outer circle) based on the first two words. we find that qameleon (figure 3(c )) generates more di-verse questions in comparison to tydiqa-goldp (figure 3(f) ). the distribution of question words varies across languages in both the datasets. for example, diversity is higher for russian, finnish, and indonesian (figure 3(a , d )); however, for bengali and telugu (figure 3(b , e )), the distribution of questions is skewed towards a specific question type ('what' for telugu and 'other' for bengali). this could be attributed to a lack of diversity in questions for these languages or poor translation quality leading to skewed utterances. multilingual qa in this work we used mt5-xl @xcite as our reference qa model. we note that a slightly more performant choice could have been byt5 @xcite , which reports improvements on tydiqa-goldp by operating directly on raw text instead of sentence pieces. existing work on low resource multilingual qa has been relatively limited. @xcite propose to use automatically translated high-confidence qa examples for training, while other approaches @xcite ) only generate questions and require supervised training data in the target language. other approaches @xcite @xcite focus on zero-shot transfer, i.e., a multilingual model trained on qa data generation on squad (and optionally automatically translated squad data) is applied to other languages. our work shows that few-shot settings result in better multilingual generation quality in comparison to zero-shot models. prompting existing work @xcite has shown that prompting pre-trained large language models can lead to strong performance in a wide range of tasks including natural language generation and common sense reasoning. in the context of multilingual qa, @xcite employ a single prompt and a few labeled examples in the target language. in contrast, we employ chainof-thought prompting, and english answers and questions as a bridge. moreover, our experiments with qameleon demonstrate that prompt tuning is superior and a viable alternative to largescale annotation. prompting in multilingual settings has achieved the best performance using english prompts and target language exemplars @xcite @xcite . we demonstrate that parameter-efficient methods such as prompt tuning using target language exemplars @xcite is a superior choice. ## benefits and limitations the method proposed in this work, qameleon, prompt tunes large plms to generate multilingual synthetic question answering data. in this section we discuss its benefits over related approaches, but also drawbacks and limitations. the main benefits are large performance improvements over alternative methods, as borne out by our experiments, as well as surprising data efficiency achieved through large-scale pre-training and a few manual annotations. alternative methods considered here are multilingual qa approaches for low resource languages, such as translate-test, translate-train, fine-tuning multilingual models directly on the small amount of available training data, performing multilingual qa directly through in-context learning, or even synthetic data generation with prompt engineered plms. another benefit of our approach stems from prompt tuning itself, which is able to learn from a tiny number of training examples, as low as one example per language in our experiments, whereas fine-tuning cannot be utilized as easily. prompt tuning also affords the practical advantage of being space efficient; a fraction of a percent of the storage space is used to save the learned parameters, since only the learned soft prompt needs to be stored. our evaluation methodology also provides benefits, since we measure question answer performance directly on downstream models instead of using a proxy like bleu or rouge on generated questions. as shown in the main drawback of our method is the high computational cost to prompt tune the plm and to generate the synthetic data. while prompt tuning is not as expensive as fine-tuning, we still need to perform optimization on a model containing hundreds of billions of parameters. we estimate the cost of each prompt tuning and data generation experiment to be in the order of 256 tpu v3 chips for 12 hours. another limitation of our experimental results is that they are fundamentally tied to a specific large plm. plms are an area of active research, so any improvements in pre-training techniques, construction of pre-training sets, instruction tuning or reinforcement learning, are likely to translate in improvements for our synthetic data generation method. promising areas of future work are parameter efficient techniques similar to prompt tuning, as well as analysis of data augmentations techniques like qameleon across different types and sizes of plms. moreover, a more formal understanding of how the number of manual annotations (aka few shots) interacts with the quality of synthetic generation, would also be useful. perhaps somewhat counter-intuitively, our experiments showed that qa performance does not drastically improve when scaling from 50 to 100 manual examples. ## data release to assist with the replicability of our results and to allow other researchers to benefit from our work, we will release a significant portion of the synthetic data generated by qameleon in the 5-shot scenario. to minimize the chance that question-answer pairs generated by the plm contain sensitive, offensive or controversial material, we vetted each generated question with three human raters. we asked each rater to discard question-answer pairs that made generalized claims about groups, contained opinions that were potentially disparaging or embarrassing to one or more people, or names of individuals not related to media (e.g., film, tv) or sport. the release will contain 47,173 examples, each with a wikipedia passage, a question and an extractive answer, corresponding to 89% of the examples utilized in this work for the 5-shot scenario. ## conclusions in this work, we examined the ability of pre-trained language models to generate synthetic data for bootstrapping multilingual qa systems, with as few as five examples in a new target language. we introduced qameleon, a parameter efficient approach which uses prompt tuning to automatically create multilingual qa data. extensive experiments under different resource scenarios demonstrate that qameleon is superior to prompt engineering and competitive baselines based on machine translation. in the future, we would like to extend this approach to other multilingual tasks, including retrieval, summarization, and semantic parsing.