Zero-shot cross-lingual open domain question answering (2022)

Abstract: People speaking different kinds of languages search for information in a cross-lingual manner. They tend to ask questions in their language and expect the answer to be in the same language, despite the evidence lying in another language. In this paper, we present our approach for this task of cross-lingual open-domain question-answering. Our proposed method employs a passage reranker, the fusion-in-decoder technique for generation, and a wiki data entity-based post-processing system to tackle the inability to generate entities across all languages. Our end-2-end pipeline shows an improvement of 3 and 4.6 points on F1 and EM metrics respectively, when compared with the baseline CORA model on the XOR-TyDi dataset. We also evaluate the effectiveness of our proposed techniques in the zero-shot setting using the MKQA dataset and show an improvement of 5 points in F1 for high-resource and 3 points improvement for low-resource zero-shot languages. Our team, CMUmQA’s submission in the MIA-Shared task ranked 1st in the constrained setup for the dev and 2nd in the test setting.

## introduction question answering (qa), especially in english, is a popular research area in nlp with abundance of datasets like squad @xcite , natural questions @xcite and different types of tasks including machine reading comprehension or extractive qa, cloze-completion and open-domain qa @xcite . open-domain qa is the task of answering natural language questions without any specified predefined context. it usually requires the system to first search for the relevant documents as the context w.r.t. a given question from either a local document repository or wikipedia-like document collection, and then generate the answer. cross-lingual open-domain question answering is a challenging nlp task, where questions are given in a user's preferred language, and the system needs to find evidence in cross-lingual large-scale document collections, like wikipedia, and return an answer in the user's preferred language, as indicated by their question. we work on this crosslingual open-domain qa challenge as a part of the mia shared task. 1 recent advancements in open-domain qa, usually for english are made by following a retriever-reader architecture, where the retriever is aimed at retrieving relevant documents w.r.t. a given question, which can be modeled as a dense passage retriever trained on large-scale english qa datasets to fetch evidence passages @xcite , while reader aims at inferring the final answer from the retrieved documents, which is usually a neural mrc model @xcite or a generative model @xcite . extending such approaches to a multilingual setting usually suffers from two major problems -1) answering questions from different language sources because the answer for low resource languages might lie in documents from high resource languages @xcite , and wikipedia which might fail in cases of same language retrieval. @xcite . 2) large-scale cross-lingual datasets are not available that supply passages in a diverse number of languages which can enable better training of cross-lingual retrievers. one specific approach that has been followed for bringing multilingual qa close to english qa is that the non-english question is translated into english and the answer from the english qa system is translated back to the query language. these systems suffer from the problem of machine translation error propagating itself in the downstream question answering. and also, these systems aren't able to exploit the fact that for high resource languages like spanish, and chinese the evidence might lie in the target language itself which is eas-ier than two-way translation. in this paper, we aim to extend the task of crosslingual question answering to tackle the following research questions -a) how can we adapt the retrieve-then-generate approaches for english open qa to cross-lingual qa that do not rely on machine translation? -b) how do multilingual qa models trained on a small set of languages perform in zero-shot settings? we follow cora @xcite which is a many-to-many multilingual qa model by following a four-stage pipeline for addressing crosslingual qa. the dpr based on mbert (mdpr) is a bi-encoder retriever that retrieves documents cross-lingually without relying on machine translation. xlm-roberta which serves as a passage reranker is trained as a cross-encoder to capture the interactions between the question and the passage on the top k documents fetched by the mdpr retriever. the reranked documents are passed through a fusion-in-decoder based mt5 reader module which can effectively learn to collect evidence from multiple passages to arrive at the final answer. in some cases, the predicted answer is not in the target language as desired by the user because the generator is either not able to convert entities into the target language or evidence is directly extracted from a different language passage. further, we use a postprocessing step to map entities from wikidata to convert the answer into the target language. we conduct our experiments on two multilingual open-domain qa datasets, xor-tydi qa @xcite and mkqa @xcite across 14 typologically diverse languages with cora as the baseline. we also use english questions from nq @xcite for training. reranking the outputs from the retriever leads to consistent improvements across all languages in both xor-tydi and mkqa, even in zero-shot settings. we show that using a fusion-indecoder based reader leads to 2.7 points improvement in em and 0.6 points improvement in f1 for the xor-tydi dataset. moreover, on applying wikidata based postprocessing techniques we see a straight 4.6 points improvement in em and 3 points in f1. we see that our proposed pipeline also helps in zero-shot settings for both high resource and low resource languages. ## datasets in this work, we use the data corpus provided by the mia shared task on cross-lingual open-retrieval qa which consists of xor-tydi and mkqa corpus. the shared task also provides questions from the nq corpus. xor-tydi is the first corpus to combine information-seeking questions, and open-retrieval qa in the multilingual domain to enable crosslingual answer retrieval. this dataset is an extension of the tydi qa @xcite dataset and involves retrieving evidence passages from multilingual and english resources. this dataset consists of questions written by native speakers in 7 typologically diverse languages: arabic, bengali, finnish, japanese, korean, russian, and telugu. mkqa corpus was originally proposed in @xcite and consists of 10k questionanswer pairs aligned across 26 typologically diverse languages (260k question-answer pairs in total). answers for this corpus are heavily curated and obtained from language-independent data representation which makes this corpus ideal for evaluating across diverse languages and being independent of language-specific passages. mkqa corpus provided by the shared task is a filtered version that only includes questions with answer annotations and removes the "no answers" questions. for this task, 12 languages were collected from mkqa, six seen: arabic, finnish, japanese, korean, russian, and six unseen(zero-shot): spanish, khmer, malay, swedish, turkish, simplified chinese, each with 1.7k examples in the dev set. nq (natural questions) is a factoid-based english question answering dataset with both short and long answers for each question from english wikipedia. we focus on the subset which is given as a part of the training dataset in the shared task. ## related work open-domain question answering requires a model to answer questions without any pre-trained domain @xcite . there have been some recent works to create a non-english qa corpus to analyze the model's effectiveness to transfer knowledge from the english language or other highresource languages. further, some works focus on generating loosely aligned data using translation or similar multilingual sources. as mentioned some of the recent works in question answering (qa) aim to build systems that can work well with languages other than english. @xcite proposed mlqa which is a multi-way aligned extractive qa corpus. it consists of instances in 7 languages with each instance parallel between 4 languages on average. this work defines two tasks: the first one focuses on analyzing the model's ability to transfer by training and testing in different languages and the other task requires the model to retrieve passages in a different language than the question. one of the shortcomings of this corpus is that it contains context in the same language and therefore doesn't explicitly captures the cross-lingual aspect. this leads to a problem for a low-resource language question set as in real scenarios most of these questions have answers in a high resource language. @xcite presents the xqa dataset to investigate crosslingual openqa research. this corpus consists of the training set in english along with the development and test set in eight other languages. their analysis of several baseline models indicates that the performance in a cross-lingual setting not only depends on the similarity of english and the target language but also on the complexity of the target language question set. another work in the crosslingual domain, xquad is proposed by @xcite which is created by using a subset of squad v1.1 @xcite corpus and translating them into ten other languages by professional translators. this paper also evaluates the hypothesis that multilingual models perform well due to the shared subword vocabulary and joint training across multiple languages and shows that monolingual representations can be adapted to produce similar performance without relying on a shared vocabulary or joint training. most previous works modeled cross-lingual qa as an extractive task which is mostly inspired by the datasets like xquad @xcite which is a subset of squad @xcite . the squad dataset contains answer spans in the evidence passage to answer a given question. these answer spans were further used in the generation of the cross-lingual qa dataset, xquad, and therefore are more suitable to be modeled as an extractive task. more recently, there have been studies that work on generating answers from raw text. works such as @xcite study cross-lingual question generation. @xcite proposed a method to generate multilingual question-answer pairs through the use of a single fine-tuned multilingual t5 generative model. their work shows that these synthetic examples could be used to improve the performance of multilingual qa in the zero-shot setting on target languages. previous works have also explored other variants of generative modeling but it was mostly limited to the domains where the model is expected to generate long answers. recent work on fid @xcite shows that generative approaches could achieve competitive results even in the cases where answers consist of a short text span. one of the widely used approaches for open-domain question answering named rag @xcite ) makes use of the generative model approach. rag model's reader module takes several retrieved passages from the retriever encoder simultaneously to generate the answer. passage representations and their similarity score with the query are used to generate the final response in the reader module. further, the rag approach works efficiently at scale due to the independent processing of passages in the encoder module. bi-encoder retrievers are effective in bringing out relevant passages from a large index but sometimes reranking those passages is essential as the downstream reader can only see a limited number of them. @xcite uses reranker after their retriever as a cross-encoder to improve the recall which proves effective in the end-to-end question answering pipeline. incorporating wikidata @xcite in translation for sentences including named entities is common in literature because the pretrained multilingual reader is unable to translate these entities as it has not seen them during training. ## baseline we use cora @xcite as our baseline model which is a unified multilingual open qa model for many languages. cora model is a combination of two models: multilingual dense passage retriever (mdpr) and multilingual answer generation (mgen). mdpr extends dense passage retriever (dpr) to a multilingual setting and uses an iterative training approach to fine-tune a pre-trained multilingual model (mbert) to encode passages and questions separately. mgen uses a multilingual sequence-tosequence model (mt5) to generate answers in the target language token-by-token given the retrieved multi-lingual passages. the generation approach is used as it can generate an answer in the target language from passages across different languages. ## methodology we employ the widely used retrieve-then-generate, figure 1 , architecture for open domain question answering. to tackle the challenge of passing a limited number of passages to the generator, we use a reranker. we also use a postprocessor to convert named entities into the query language. ## results & discussion for xor-tydi, we see that overall we achieve 3 and 4.6 points improvement in f1 and em respectively, whereas for mkqa we achieve a 5.9 points f1 improvement. we see that adding fid to the baseline mdpr leads to a significant increase in f1 for languages like ar, fi, and ru. further, adding postprocessing to the fid output leads to a significant increase in both f1 and em and the model outperforms the baseline for all the languages. reranking is crucial for fid because it sees only 50 documents as compared to the 100 that the baseline uses. reranking the top 200 documents retrieved helps the resulting fid which shows a consistent improvement over the baseline except for bn and ko for which the model usually outputs lots of entities in english that require post-processing. we get the best results for applying postprocessing (p) on the outputs by the reranker(rr) + fid model. for zero-shot settings in the mkqa dataset, we also see consistent 5 points of f1 improvement over the baseline due to the combined effect of the fid, reranker, and the postprocessor. for high-resource zero-shot languages es, sv, zh we observe around 5 points improvement over baseline whereas for km which is an extremely low-resource language we still show around 3 points improvement. we now provide detailed results and analysis for each component of our pipeline. ## conclusion & future work we introduced a modular end-to-end system with a retriever, reranker, reader, and postprocessor for cross-lingual question answering and showed improvements in both normal and zero-shot settings. these cross-lingual models consist of a large number of parameters and are very resourceintensive. the retriever model takes around 1 hr/epoch whereas the fusion-in-decoder model takes 8 hr/epoch on a6000 gpus. for future work, we think it would be interesting to try sparse retrieval methods @xcite in crosslingual settings and also try incorporating more knowledge from wikidata based entities in the pipeline.