Cross-Lingual Open-Domain Question Answering with Answer Sentence Generation (2022)

Abstract: Open-Domain Generative Question Answering has achieved impressive performance in English by combining document-level retrieval with answer generation. These approaches, which we refer to as GenQA, can generate complete sentences, effectively answering both factoid and non-factoid questions. In this paper, we extend to the multilingual and cross-lingual settings. For this purpose, we first introduce GenTyDiQA, an extension of the TyDiQA dataset with well-formed and complete answers for Arabic, Bengali, English, Japanese, and Russian. Based on GenTyDiQA, we design a cross-lingual generative model that produces full-sentence answers by exploiting passages written in multiple languages, including languages different from the question. Our cross-lingual generative system outperforms answer sentence selection baselines for all 5 languages and monolingual generative pipelines for three out of five languages studied.

## introduction improving coverage of the world's languages is essential for retrieval-based question answering (qa) systems to provide a better experience for non-english speaking users. one promising direction for improving coverage is multilingual, multisource, open-domain qa. multilingual qa systems include diverse viewpoints by leveraging answers from multiple linguistic communities. further, they can improve accuracy, as all facets necessary to answer a question are often unequally distributed across languages on the internet @xcite . with the advance of large-scale language models, multilingual modeling has made impressive progress at performing complex nlp tasks without requiring explicitly translated data. building on pre-trained language models @xcite @xcite , it is now possible to train models that accurately process textual data in multiple languages @xcite and perform crosslingual transfer @xcite using annotated data in one language to process another language. at the same time, answer generation-based approaches have been shown to be effective for many english qa tasks, including machine reading (mr) @xcite , question-based summarization @xcite @xcite , and, most relevant to this work, answer generation for retrieval-based qa @xcite ) -that we refer to as genqa. compared to generative mr models, genqa approaches are trained to produce complete and expressive sentences that are easier to understand than extracted snippets @xcite . most importantly, they are trained to generate entire sentences, allowing them to answer both factoid or non-factoid questions, e.g., asking for descriptions, explanation, or procedures. in this paper, we study and propose a simple technique for open-domain qa in a cross-lingual setting. following @xcite (and as illustrated in figure 1 ), we work with a pipeline made of 3 main modules. first, a document retriever that retrieves relevant documents given a question; second, an answer sentence selection (as2) model @xcite that ranks the sentences from the retrieved documents based on how likely they are to include the answer; and third, a generative model that generates a full sentence to answer the question given the sentence candidates. our contribution focuses on the generative model. we introduce crossgenqa. cross-genqa can generate full-sentence answers using sentence candidates written in multiple languages including languages different from the question and english. given the scarcity of annotated corpora for genqa, especially in languages different from english, we introduce the gen-tydiqa dataset. gen-tydiqa is an extension of tydiqa, a dataset for typologically diverse languages in which questions are answered with passages and short spans extracted from wikipedia @xcite . our gen-tydiqa includes humangenerated, fluent, self-contained answers in arabic, bengali, english, russian and japanese, making it a valuable resource for evaluating multilingual generative qa systems. we found human-generated answers to be essential in evaluating genqa: compared to the standard approach of providing reference documents, they dramatically speed-up annotations and improve inter-annotator agreement. our evaluation shows that our crossgenqa system outperforms as2 ranking models, and matches or exceeds similar monolingual pipelines. in summary, our contribution is three-fold: hsu et al. (2021) by showing that monolingual generative qa (monogenqa) outperforms extractive qa systems in arabic, bengali, english and russian. (iii) we demonstrate that crossgenqa outperforms all our qa systems for arabic, russian, and japanese, answering questions using information from multiple languages. 2 related work multilingual datasets for qa researchers have introduced several datasets for qa in multiple languages. unlike our gen-tydiqa, to the best of our knowledge, they are designed exclusively for extractive qa. artetxe et al. (2019) extended the english machine reading squad dataset (rajpurkar et al., 2016) by translating the test set to 11 languages. similarly, lewis et al. (2020a) collected new question and answer pairs for 7 languages following the squad format. recently, longpre et al. (2020) released mkqa, which includes question and answer pairs (predominantly yes/no answers and entities) for 26 languages. clark et al. (2020) released tydiqa, a dataset for extractive qa in 11 typologically diverse languages. riabi et al. (2020) and shakeri et al. (2021) have explored the use of techniques to synthetically generate data for extractive question answering using cross-lingual transfer. generating fluent answers for qa the generation of fluent and complete-sentence answers is still in its infancy, as most generative models for qa are used for extractive qa (e.g., @xcite @xcite . approaches to ensure response fluency have been explored in the context of dialogue systems @xcite , but remain nevertheless understudied in the context of qa. providing natural sounding answers is a task of particular interest to provide a better experience for users of voice assistants. one resource for this task is the ms-marco dataset @xcite . it includes 182,669 question and answer pairs with human-written well-formed answers. however, it only contains samples in english. our gen-tydiqa extends tydiqa @xcite ) adding natural human-generated answers for arabic, bengali, english, japanese, and russian. to the best of our knowledge, it is the first work that provides well-formed, natural-sounding answers for non-english languages. multilingual extractive qa designing qa models for languages different from english is challenging due to the limited number of resources and the limited size of those datasets. for this reason, many studies leverage transfer learning across languages by designing systems that can make use of annotated data in one language to model another language. for instance, @xcite showed that concatenating the training data from multiple languages improves the performance of a model on all the target languages for extractive qa. in the open-retrieval qa setting, multilingual modeling can be used to answer questions in one language using information retrieved from other languages. @xcite showed how cross-language tree kernels can be used to rank english answer candidates for arabic questions. @xcite designed a cross-lingual question similarity technique to map a question in one language to a question in english for which an answer has already been found. @xcite showed that extracting relevant passages from english wikipedia can deliver better answers than relying only on the wikipedia corpora of the question language. @xcite showed how machine translated questionanswer pairs can be used to train a multilingual qa model; in their study, they leveraged english data to train an english and german as2 model. finally, @xcite introduced cora and reached state-of-the-art performance on openretrieval span-prediction question answering across 26 languages. while related to our endeavor, it is significantly different in several key aspects. first, unlike crossgenqa, cora does not produce full, complete sentences; rather, it predicts spans of text that might contain a factoid answer. second, it mainly relies on sentence candidates that are written in english and in the question language; by contrast, in our work we choose to translate the questions into a variety of languages, allowing us to use monolingual retrieval pipelines to retrieve candidate sentences in diverse languages. we show that this form of cross-lingual genqa outperforms monolingual genqa in a majority of the languages studied. ## the gen-tydiqa dataset to more efficiently evaluate our multilingual generative pipeline (lower cost and higher speed), we built gen-tydiqa, an evaluation dataset for answer-generation-based qa in arabic, bengali, english, japanese, and russian. this extends the tydiqa @xcite dataset. tydiqa is a qa dataset that includes questions for 11 typologically diverse languages. each entry is composed of a human-generated question and a single wikipedia document providing relevant information. for a large subset of its questions, tydiqa also contains a human-annotated passage extracted from the wikipedia document, as well as a short span of text that answers the question. we extend the tydiqa validation set foot_1 by collecting human-generated answers based on the provided questions and passages using amazon mechanical turk foot_2 (cf. appendix c.1 for hiring criteria and rewards). collecting human-generated answers is crucial for properly evaluating genqa models, as we will show in section 5.4. we use a two-stage data collection process: ## multilingual genqa systems our goal is to build a qa system that, given a question in a target language, retrieves the top-k most relevant passages from text sources in multiple languages, and generates an answer in the target language from these passages (even if the passages are in a different language from the question). ## experiments multilinguality and the different components of our system pipeline raise interesting research questions. our experimental setup is defined by the combinations of our target set of languages with respect to questions, candidates, and answers. we experiment with genqa in the monolingual (one model per language) and multilingual (one model for several languages) settings, where the question and candidates in the same language are used to generate an answer. then we experiment with a cross-lingual genqa model that is fed candidates in multiple languages. despite being an apparent more complex task, we find that in many cases, the cross-lingual model outperform all other settings. ## limits our system requires translating the questions. we also use the standard bm25 approach. even though it was shown to be more robust compared to dense retriever @xcite , using a cross-lingual retriever @xcite could improve performance and save the cost of translating the question. this has been explored by @xcite but their retriever mainly retrieves passages in english and the question language which may lead to english-centric answers. another limit is the fact that our system is not designed to handle questions that are not answerable. in the future, we may want to integrate a no-answer setting to avoid unwanted answer. ## conclusion we study retrieval-based question answering systems using answer generation in a multilingual context. we proposed (i) gen-tydiqa, a new multilingual qa dataset that includes natural and complete answers for arabic, bengali, english, japanese, and russian; based on this dataset (ii) the first multilingual and cross-lingual genqa retrieval-based systems. the latter can accurately answer questions in one language using information from multiple languages, outperforming answer sentence selection baseline for all languages and monolingual pipeline for arabic, russian, and japanese.