Synthetic Data Augmentation for Zero-Shot Cross-Lingual Question Answering (2021)

Abstract: Coupled with the availability of large scale datasets, deep learning architectures have enabled rapid progress on the Question Answering task. However, most of those datasets are in English, and the performances of state-of-the-art multilingual models are significantly lower when evaluated on non-English data. Due to high data collection costs, it is not realistic to obtain annotated data for each language one desires to support. We propose a method to improve the Cross-lingual Question Answering performance without requiring additional annotated data, leveraging Question Generation models to produce synthetic samples in a cross-lingual fashion. We show that the proposed method allows to significantly outperform the baselines trained on English data only. We report a new state-of-the-art on four datasets: MLQA, XQuAD, SQuAD-it and PIAF (fr).

## introduction question answering is a fast-growing research field, aiming to improve the capabilities of machines to read and understand documents. significant progress has recently been enabled by the use of large pre-trained language models @xcite , which reach human-level performances on several publicly available benchmarks, such as squad @xcite and newsqa @xcite . given that the majority of large scale question answering (qa) datasets are in english @xcite @xcite , the development of qa systems targeting other languages is currently addressed via two cross-lingual qa datasets: xquad @xcite and mlqa @xcite , covering respectively 10 and 7 languages. due to the cost of annotation, both are limited only to an evaluation set. they are comparable to the validation set of the original squad (see more details in section 3.3). in both datasets, each paragraph is paired with questions in various languages, allowing to evaluate models in a cross-lingual experimental scenario: the input context and the question can be in two different languages. this scenario has important practical applications, such as querying a set of documents in various languages. performing this cross-lingual task is complex and remains challenging for current models, assuming only english training data: transfer results are shown to rank behind training-language performance @xcite . in other words, multilingual models fine-tuned only on english data are found to perform significantly better on english than on other languages. besides the almost simultaneous work of @xcite , very few alternatives to such a simple zeroshot transfer method have been proposed so far. in this paper, we propose to generate synthetic data in a cross-lingual fashion, borrowing the idea from monolingual qa research efforts @xcite . on english corpora, generating synthetic questions has shown to significantly improve the performance of qa models @xcite @xcite . however, the adaptation of this technique to cross-lingual qa is not straightforward: crosslingual text generation is a challenging task per se which has not been yet extensively explored, in particular when no multilingual training data is available. we explore two question generation scenarios: (i) requiring only squad data; and (ii) using a translator tool to obtain translated versions of squad. as expected, the method leveraging on a translator has shown to perform the best. leveraging on such synthetic data, our best model obtains significant improvements on xquad and mlqa over the state-of-the-art for both exact match and f1 scores. in addition, we evaluate the qa models on languages not seen during training (even for the synthetic data) -using squad-it (for italian), piaf (for french), and korquad (for korean) -reporting a new state-of-the-art for italian and french, and observing significant improvements on korean compared to zero-shot without augmentation. this indicates that the proposed method allows to capture better multilingual representations beyond the training languages. our method paves the way toward multilingual qa domain adaptation, especially for under-resourced languages. our contributions can be summarized as follows: question answering based on synthetic question generation; • we report extensive experiments showing significant improvements on two multilingual evaluation datasets (xquad and mlqa); • we additionally evaluate the proposed methodology on languages unseen during training, thus showing the potential benefits for qa on low-resource languages. 2 related work question answering (qa) qa is the task for which given a context and a question, a model has to find the answer. the interest for question answering goes back a long way: in a 1965 survey, simmons (1965) reported fifteen implemented english language question-answering systems. more recently, with the rise of large scale datasets (hermann et al., 2015), and large pre-trained models (devlin et al., 2019), the performance drastically increased, approaching human-level performance on standard benchmarks -see for instance the squad leader board. foot_0 more challenging evaluation benchmarks have recently been proposed: dua et al. (2019) released the drop dataset, for which the annotators were encouraged to provide adversarial questions; burchell et al. (2020) released the msq dataset, consisting of multi-sentence questions. however, all these works are focused on english. another popular research direction focuses on the development of multilingual qa models. for this purpose, the first step has been to provide the community with multilingual evaluation sets: artetxe et al. (2020) and lewis et al. (2020a) concurrently proposed two different evaluation sets which are comparable to the squad development set. both reach the same conclusion: due to the lack of non-english training data, models do not achieve the same performance in non-english languages than they do in english. to the best of our knowledge, no method has been proposed to fill this gap. question generation (qg) qg can be seen as the dual task of qa: the input is composed of the answer and the paragraph containing it, and the model is trained to generate the question. proposed by @xcite , it has leveraged on the development of new qa datasets @xcite . similar to qa, significant performance improvements have been obtained using pre-trained language models @xcite . still, due to the lack of multilingual datasets, most previous works have been limited to monolingual text generation. we note the exceptions of @xcite and @xcite , who resorted to multilingual pre-training before fine-tuning on monolingual downstream nlg tasks. however, the quality of the generated questions is still found inferior to the corresponding english ones. ## models recent works @xcite have shown that classification tasks can be framed as a text-to-text problem, achieving stateof-the-art results on established benchmarks, such as glue @xcite . accordingly, we employ the same architecture for both question answering and generation tasks. this also allows fairer comparisons for our purposes, by removing differences between qa and qg architectures and their potential impact on the results obtained. in particular, we use a distilled version of xlm-r @xcite : minilm-m @xcite (see section 4.3 for further details). ## conclusion in this work, we presented a method to generate synthetic qa dataset in a multilingual fashion, showing how qa models can benefit from it and reporting large improvements over the baselines. the proposed approach contributes to fill the gap between english and other languages, and is shown to generalize for languages not present in the synthetic corpus (e.g. french, italian, korean). in future work, we plan to investigate whether the proposed data augmentation method could be applied to other multilingual tasks, such as classification. we will also experiment more in depth with different strategies to control the target language of a model, and extrapolate on unseen ones. in addition, we report the f1 scores for xlm-r finetuned on squad en and xlm-r +synth-trans on all the language pairs, on both mlqa and xquad in tables 7 , 8 , 9 and 10.