PAXQA : Generating Cross-lingual Question Answering Examples at Training Scale (2023)

Abstract: Existing question answering (QA) systems owe much of their success to large, high-quality training data. Such annotation efforts are costly, and the difficulty compounds in the cross-lingual setting. Therefore, prior cross-lingual QA work has focused on releasing evaluation datasets, and then applying zero-shot methods as baselines. This work proposes a synthetic data generation method for cross-lingual QA which leverages indirect supervision from existing parallel corpora. Our method termed PAXQA (Projecting annotations for cross-lingual (x) QA) decomposes cross-lingual QA into two stages. First, we apply a question generation (QG) model to the English side. Second, we apply annotation projection to translate both the questions and answers. To better translate questions, we propose a novel use of lexically-constrained machine translation, in which constrained entities are extracted from the parallel bitexts. We apply PAXQA to generate cross-lingual QA examples in 4 languages (662K examples total), and perform human evaluation on a subset to create validation and test splits. We then show that models fine-tuned on these datasets outperform prior synthetic data generation models over several extractive QA datasets. The largest performance gains are for directions with non-English questions and English contexts. Ablation studies show that our dataset generation method is relatively robust to noise from automatic word alignments, showing the sufficient quality of our generations. To facilitate follow-up work, we release our code and datasets at https://github.com/manestay/paxqa.

## introduction a common framing of question answering (qa) in nlp is as a reading comprehension task, where questions about a specific text are to be answered by a span from a given context. developing strong qa systems thus advances progress towards developing systems which can read and reason about texts. while earlier work developed qa models and resources in english only @xcite , recent work has sought to extend beyond english. such datasets include tydi qa (11 languages; @xcite ), mlqa (6 languages; @xcite , xquad (10 languages; @xcite , and mkqa (26 languages; @xcite , and are annotated with the help of native speakers of diverse languages. however, the high annotation cost required means they are limited to evaluation, and there is no data available for training. these works therefore use several zero-shot approaches as baselines on their datasets. first, zeroshot transfer involves fine-tuning a multilingual pre-trained language model (lm) on english qa data, then applying this model directly to multilingual qa. another, translate-train, uses a machine translation system (mt) to translate english data into other languages, then train new models on the translated data. third, translate-test instead uses an english qa model, at inference time translating other language qa to english, then back for the final evaluation. alternatively, recent work has shown promising results with synthetic data augmentation @xcite @xcite . in this approach, a question generation (qg) model is trained to generate synthetic multilingual qa examples, which are used as training data for a downstream qa model. in this work, we propose a methodology for synthetic data augmentation, motivated by the insight that generating cross-lingual qa need not be entirely zero-shot. instead, indirect supervision can be taken from widely-available parallel datasets originally collected for machine translation. our method termed paxqa (projecting annotations for cross-lingual (x) qa) extracts qa examples from figure 1 : the paxqa method generates a cross-lingual question-answering (qa) dataset given a word-aligned and parallel corpus. the two stages are english question generation (left), and question and answer translation (right). we run the pipeline on {ar-en}, {zh-en}, and {ru-en} datasets (bottom), resulting in 662k cross-lingual qa examples -usable at training scale. our generation pipeline proceeds similarly to prior works' annotation pipelines, but our method replaces all instances of human annotation with automated methods. parallel corpora. paxqa decomposes the task into two stages: english question and answer generation, then machine translation of questions and answers informed by word alignments. through this decomposition, paxqa serves as a framework in which the individual qg and mt systems can be updated with the latest developments. we apply our methodology to generate largescale cross-lingual qa datasets in 4 languages: chinese, arabic, russian, and english. we validate the quality of our generations by showing both improvements for downstream qa tasks, and by performing a human evaluation task. our four key contributions are: ## task definition in this work, we focus on cross-lingual extractive question answering. a cross-lingual extractive qa dataset consists of qa entries, each of which contains a context c f , an answer a f , and a question q e (where f and e denote a source and a target language). the task is defined as follows: given q e and c f , a model must output an a f which is extracted from c f . our goal is to both propose a method to synthetically generate such a dataset, and to train a model to solve the task. following prior works' broad definition of crosslingual extractive qa @xcite , it is possible that the two languages are the same (@xmath0. the only restriction is that in order to ensure the qa task is extractive, the context and answer must be in the same language; the question may or may not be in the same language. in the literature, multilingual qa foot_1 most commonly refers to the setting where qa is monolingual, but in multiple languages @xcite . while the dataset as a whole considers multiple languages l n 1 = l 1 , ...l n ., each entry on its own is monolingual -(c, q, a) l i , where l @xmath1. a crosslingual qa dataset, in contrast, includes both monolingual and multilingual entries. notation we denote a language pair as {f -e}. a language pair covers 4 cross-lingual directions (f, e; e, f ; f, f ; e, e). for each direction, the elements represent the language of the context and of the question, respectively. for example, f, e means a context in f , and a question in e. ## data generation method we now describe the paxqa data generation methodology, which was outlined in figure 1 . the goal is to generate synthetic cross-lingual qa datasets den,l , for each language of interest l (ŝ pecifies that it is generated). our method assumes the availability of a parallel corpus p en,l between english (en) and each l. we denote the two sides of a parallel corpus as p en and p l . we also assume that these parallel corpora are word-aligned, either by humans or by a word alignment tool. we now describe the two stages of our methodology. ## experimental setup although the paxqa method can be applied to any language l with english-l parallel data, in our experiments we address three languages: chinese (zh), arabic (ar), and russian (ru). we hope that the diversity in scripts and language families will illustrate the wide applicability of our method. ## results the paxqa generation method creates synthetic qa data, we can then be used to fine-tune a crosslingual extractive qa model. qa performance that beats prior work is our first way of validating the quality of the generations. we report results on three datasets: paxqa hwa test, mlqa @xcite , and xorqa goldp @xcite . the evaluation metric is the mean token f1 score, calculated with the official mlqa script. 7 prior work reports average results across all cross-lingual directions. we instead group directions as follows: 'non-en q'uestion + en context; 'en q'uestion + non-en context; 'monolingual', and 'non-en (xling)' cross-lingual. we then analyze the results by group, which gives us a clearer picture of how the methods affect different directions. on mlqa, we discuss results only in {ar,zh,en}; results averaged over all 49 mlqa directions are in appendix qa model we adopt xlm-r @xcite ) as our qa model, and initialize to the pretrained large checkpoint from the transformers library @xcite . following the advice of @xcite , we fine-tune in two rounds: first on synthetic qa data, then on squad. we find that this improves results over shuffling real and synthetic data. ## conclusion we present paxqa, a synthetic data generation methodology for cross-lingual qa which leverages indirect supervision from parallel datasets. we decomposed the task into two stages: english qa generation, then qa translation informed by annotation projection. unlike prior methods, paxqa neither requires training of new models, nor non-english qa data. this means our method can even be applied to low-resource languages. we proposed the novel use of lexically-constrained mt to better translate questions, which assists in proper translation of uncommon entities. finally, we showed that training on paxqa data allows downstream models to significantly outperform baselines, and achieve a new zero-shot state-of-the-art on the mlqa benchmark. we close by reiterating that paxqa is a methodology, and that as such, the machine components for mt and word alignment can be swapped out as new developments progress in those fields.