Towards Zero-Shot Multilingual Synthetic Question and Answer Generation for Cross-Lingual Reading Comprehension (2021)

Abstract: We propose a simple method to generate multilingual question and answer pairs on a large scale through the use of a single generative model. These synthetic samples can be used to improve the zero-shot performance of multilingual QA models on target languages. Our proposed multi-task training of the generative model only requires labeled training samples in English, thus removing the need for such samples in the target languages, making it applicable to far more languages than those with labeled data. Human evaluations indicate the majority of such samples are grammatically correct and sensible. Experimental results show our proposed approach can achieve large gains on the XQuAD dataset, reducing the gap between zero-shot and supervised performance of smaller QA models on various languages.

## introduction generating question and answers from raw text has always been a challenging problem in natural language generation. recently, there have been numerous efforts around question generation @xcite @xcite @xcite @xcite . using such synthetic samples to improve the performance of question answering models has been explored by @xcite , @xcite , and @xcite , who show that reading comprehension (rc) models can be improved by generating large-scale synthetic training data. these promising results combined with the recent surge in the development of powerful generative models such as gpt-3 @xcite , bart @xcite , and t5 @xcite 2) generating synthetic samples from wikipedia passages of the target language using the fine-tuned generative model. in this work, we propose generating multilingual question answer pairs to improve the performance of rc models in languages other than english. besides unlabeled articles and questions, our proposed method only requires labeled training samples in english, thus completely removing the need to acquire new labeled datasets. our approach can easily be extended to any language, as long as the multilingual generative model supports the language, and unlabeled questions and articles, such as wikipedia, books, etc., exist in that language. to enable zero-shot qa generation, the generative model should be able to produce non-english qa samples from non-english inputs when only trained on english samples. inspired by the work of @xcite ; @xcite ; @xcite , we propose a multi-task learning setting, where during the fine-tuning stage, we train on two tasks in parallel: the target question-answer generation task, and the multilingual masked language modeling (mlm) task that was used in pretraining the generative model. our experimental results show that including the mlm task is crucial in enabling the zero-shot capability of the fine-tuned generative model. we propose fine-tuning a pre-trained multilingual t5 model on the squad 1.1 @xcite training set. the fine-tuned model is then used to generate a large set of synthetic questionanswer pairs from wikipedia passages in the target language. fig. 1 illustrates the end-to-end pipeline. we show that such synthetic samples can significantly boost rc models trained only on the english samples, with improvements up to 9 absolute points on f1. to summarize, our contributions are: the rest of the paper is organized as follows. in section 2, we discuss the process designed to train the generative model and produce synthetic samples. section 3 discusses related work in the area of multilingual question-answer generation. in section 4, we present experiments to measure the quality of generated samples. section 5 focuses on the application of synthetic question-answer samples to downstream reading comprehension models. finally, we conclude in section 6. ## related work recent work has explored question-answer generation @xcite @xcite , but limited in scope to english. we leverage the modeling and filtering approaches proposed by @xcite due to their simplicity and effectiveness. @xcite explores cross lingual question generation. in contrast to our work, this only generates questions, without the corresponding answers. additionally, this approach requires a complicated pre-training process on the target languages, as well as gold samples to fine-tune the generative models, so it is not easily extensible to other languages. this is in contrast to our approach, which does not require any gold qa samples in any language other than english. another distinguishing factor is that we demonstrate improved performance on downstream qa tasks, while @xcite only measure the quality of the generated samples on automatic metrics such as bleu, and human evaluations. similarly, @xcite explore cross-lingual question-only generation using squad english samples. they propose cross-lingual pre-training on the source and target languages. similar to kumar et al. (2019) , their focus is only on the quality of the generated questions, whereas we validate our approach directly through improvements on downstream qa tasks. moreover, while chi et al. (2020) depends on a complex pre-training recipe and parallel sentences in both source and target languages, our approach not only does not require such parallel corpus, but also the mlm task included in our fine-tuning setting is widely used and studied. this leads to our approach being more easily adaptable to other languages and pre-trained generative models. most closely related to our work is the multilingual synthetic question generation approach of @xcite . however, there are two important differences between the two approaches. firstly, our work includes both question and answer generation using a single model, while theirs only focuses on question generation. we believe generating the question and answer jointly is a richer problem that better harnesses the capabilities of pre-trained language models. their question generation is conditioned on the selected answers, which further limits the generation. secondly, their proposed method depends on translating squad to target languages to fine-tune the generative model, hence limiting the application of their approach to languages where such translation data exists. furthermore, even when translated data is available, the quality of samples generated by a model trained on such data is highly affected by the quality of the translations. this could lead to low quality qa samples in low resource languages. this is in contrast to our zero-shot approach, which does not require any training data in the target language. ## application of synthetic data to multilingual reading comprehension in this section, we describe experimental results that demonstrate the efficacy of using synthetic samples for improving multilingual reading comprehension (rc) models. this refers to the setting where given a passage and a question, the model is tasked with finding a span of the passage that answers the question. ## ethical considerations since the synthetic qa samples are generated by a generative model, it is possible that generated questions could include hallucinations and counterfactual information. we have employed the following safeguards: 1) the generative model is trained on the squad dataset to learn question-answer generation. squad is a well-studied and meticulously curated dataset. 2) the passages from which question-answer pairs are generated are selected from wikipedia. 3) we apply round-trip filter on the generated question-answer pairs using the rc model. this approach ensures the questions are relevant to the passages. we believe these steps drastically reduce the chances of hallucinated and counterfactual samples. nevertheless, there still exists the possibility that such bad samples could be generated. future research efforts can explore such potential issues.