Z usammen QA : Data Augmentation with Specialized Models for Cross-lingual Open-retrieval Question Answering System (2022)

Abstract: This paper introduces our proposed system for the MIA Shared Task on Cross-lingual Openretrieval Question Answering (COQA). In this challenging scenario, given an input question the system has to gather evidence documents from a multilingual pool and generate from them an answer in the language of the question. We devised several approaches combining different model variants for three main components: Data Augmentation, Passage Retrieval, and Answer Generation. For passage retrieval, we evaluated the monolingual BM25 ranker against the ensemble of re-rankers based on multilingual pretrained language models (PLMs) and also variants of the shared task baseline, re-training it from scratch using a recently introduced contrastive loss that maintains a strong gradient signal throughout training by means of mixed negative samples. For answer generation, we focused on languageand domain-specialization by means of continued language model (LM) pretraining of existing multilingual encoders. Additionally, for both passage retrieval and answer generation, we augmented the training data provided by the task organizers with automatically generated question-answer pairs created from Wikipedia passages to mitigate the issue of data scarcity, particularly for the low-resource languages for which no training data were provided. Our results show that language- and domain-specialization as well as data augmentation help, especially for low-resource languages.

## introduction open-retrieval question answering (oqa), where the agent helps users to retrieve answers from largescale document collections with given open ques-tions, has arguably been one of the most challenging natural language processing (nlp) applications in recent years (e.g., @xcite @xcite . as is the case with the vast majority of nlp tasks, much of the oqa focused on english, relying on a pipeline that crucially depends on a neural passage retriever, i.e., a (re-)ranking model -trained on large-scale english qa datasets -to find evidence passages in english @xcite for answer generation. unlike in many other retrieval-based tasks, such as ad-hoc document retrieval @xcite , parallel sentence mining @xcite , or entity linking @xcite , the progress toward cross-lingual open-retrieval question answering (coqa) has been hindered by the lack of efficient integration and consolidation of knowledge expressed in different languages @xcite . coqa is especially relevant for opinionated information, such as news, blogs, and social media. in the era of fake news and deliberate misinformation, training on only (or predominantly) english texts is more likely to lead to more biased and less reliable nlp models. further, an anglo-and indo-european-centric nlp @xcite is unrepresentative of the needs of the majority of the world's population (e.g., mandarin and spanish have more native speakers than english, and hindi and arabic come close) and contributes to the widening of the digital language divide. foot_0 developing solutions for cross-lingual open qa (coqa) thus contributes towards the goal of global equity of information access. coqa is the task of automatic question answering, where the answer is to be found in a large ## data augmentation we use a language model to generate questionanswer (qa) pairs from english texts, which we then filter according to a number of heuristics and translate into the other 15 languages. 2 an example can be seen in figure 2 in the appendix. ## methodology following the approach described in @xcite , we consider the coqa problem as two sub-components: the retrieval of documents containing a possible answer and the generation of an answer. figure 1 summarizes the proposed methods. this section is organized as follows: we present the proposed retrieval methods in §3.1 and demonstrate the language-specialized methods for answer generation in §3.2. ## experimental setup we demonstrate the effectiveness of our proposed coqa systems by comparing them to the baseline models and thoroughly comparing different specialization methods from §1. evaluation task and measures our proposed approaches are evaluated in 16 languages, 8 of which are not covered in the training data. 5 the training and evaluation data are originally from natural questions @xcite , xor-tydi qa @xcite , and mkqa @xcite . data size statistics for each resource and language are shown in the evaluation results are measured on the competition platform hosted at eval.ai. 6 the systems are evaluated on two coqa datasets: xor-tydi qa @xcite , and mkqa @xcite , using token-level f1 (f1), as common evaluation practice of open qa systems @xcite . for non-spacing languages, we follow the token-level tokenizers foot_6 for both predictions and ## results and discussion results overview results in unsupervised vs supervised retrieval in all evaluation settings, unsupervised retrieval methods underperform supervised methods by a large margin (see ## related work passage retrieval and answer generation to improve the information accessibility, openretrieval question answering systems are attracting much attention in nlp applications @xcite . @xcite were one of the early works to present a benchmark that requires systems to understand a passage to produce an answer to a given question. @xcite presented a more challenging and realistic dataset with questions collected from a search engine. to tackle these complex and knowledge-demanding qa tasks, lewis et al. (2020) proposed to first retrieve related documents from a given question and use them as additional aids to predict an answer. in particular, they explored a general-purpose fine-tuning recipe for retrieval-augmented generation models, which combine pretrained parametric and non-parametric memory for language generation. izacard and grave (2021), they solved the problem in two steps, first retrieving support passages before processing them with a seq2seq model, and @xcite further extended to the cross-lingual conversational domain. some works are explored with translatethen-answer approach, in which texts are translated into english, making the task monolingual @xcite . while this approach is conceptually simple, it is known to cause the error propagation problem in which errors of the translation get amplified in the answer generation stage @xcite . to mitigate this problem, @xcite proposed to extend lewis et al. (2020) by using multilingual models for both the passage retrieval @xcite and answer generation @xcite . data augmentation data augmentation is a common approach to reduce the data sparsity for deep learning models in nlp @xcite . for question answering (qa), data augmentation has been used to generate paraphrases via backtranslation @xcite , to replace parts of the input text with translations @xcite , and to generate novel questions or answers @xcite @xcite . in the cross-lingual setting, available data have been translated into different languages @xcite @xcite and language models have been used to train question and answer generation models @xcite @xcite . our approach is different from previous work in cross-lingual question answering task in that it only requires english passages to augment the training data, as answers are generated automatically from the trained model by @xcite . in addition, our filtering heuristics remove incorrectly generated question-answer pairs, which allows us to keep only question-answer pairs with answers that are more likely to be translated correctly, thus limiting the problem of error propagation. ## reproducibility to ensure full reproducibility of our results and further fuel research on coqa systems, we release the model within the huggingface repository as the publicly available multilingual pretrained language model specialized in 14 and 16 languages. 11 we also release our code and data, which make our approach completely transparent and fully reproducible. all resources developed as part of this work are publicly available at: https: //github.com/umanlp/zusammenqa . ## conclusion we introduced a framework for a cross-lingual open-retrieval question-answering system, using data augmentation with specialized models in a constrained setup. given a question, we first retrieved top relevant documents and further generated the answer with the specialized models (i.e., mlm-ing on wikipedia passages) along with the augmented data variants. we demonstrated the effectiveness of data augmentation techniques with language-and domain-specialized additional training, especially for resource-lean languages. however, there are still remaining challenges, especially in the retrieval model training with limited computational resources. our future efforts will be to focus on more efficient approaches of both multilingual passage retrieval and multilingual answer generation @xcite with the investigation of different data augmentation techniques @xcite . we hope that our generated qa language resources with the released models can catalyze the research focus on resource-lean languages for coqa systems.