AfriQA : Cross-lingual Open-Retrieval Question Answering for A frican Languages (2023)

Abstract: African languages have far less in-language content available digitally, making it challenging for question answering systems to satisfy the information needs of users. Cross-lingual open-retrieval question answering (XOR QA) systems – those that retrieve answer content from other languages while serving people in their native language—offer a means of filling this gap. To this end, we create Our Dataset, the first cross-lingual QA dataset with a focus on African languages. Our Dataset includes 12,000+ XOR QA examples across 10 African languages. While previous datasets have focused primarily on languages where cross-lingual QA augments coverage from the target language, Our Dataset focuses on languages where cross-lingual answer content is the only high-coverage source of answer content. Because of this, we argue that African languages are one of the most important and realistic use cases for XOR QA. Our experiments demonstrate the poor performance of automatic translation and multilingual retrieval methods. Overall, Our Dataset proves challenging for state-of-the-art QA models. We hope that the dataset enables the development of more equitable QA technology.

## introduction question answering (qa) systems provide access to information @xcite and increase accessibility in a range of domains, from healthcare and health emergencies such as covid-19 @xcite to legal queries @xcite and financial questions @xcite . many of these applications are particularly important in regions where information and services may be less accessible and where language technology may thus help to reduce the burden on the existing system. at the same time, many people prefer to access information in their local languages-or simply do not speak a dataset qa? clir? open retrieval? # languages # african languages xqa @xcite ✓ ✓ ✓ 9 nil xor qa @xcite ✓ ✓ ✓ 7 nil xquad @xcite ✓ ✗ ✗ 11 nil mlqa @xcite ✓ ✗ ✗ 7 nil mkqa @xcite ✓ ✗ ✓ 26 nil tydi qa @xcite ✓ ✗ ✓ 11 1 amqa @xcite ✓ ✗ ✗ 1 1 kenswquad @xcite ✓ ✗ ✗ 1 1 afriqa (ours) ✓ ✓ ✓ 10 10 (see standard qa datasets mainly focus on english @xcite @xcite . while some reading comprehension datasets are available in other high-resource languages @xcite , only a few qa datasets @xcite @xcite cover a typologically diverse set of languages-and very few datasets include african languages (see in this work, we lay the foundation for research on qa systems for one of the most linguistically diverse regions by creating afriqa, the first qa dataset for 10 african languages. afriqa focuses on open-retrieval qa where information-seeking questions 2 are paired with retrieved documents in which annotators identify an answer if one is available @xcite . as many african languages lack high-quality in-language content online, afriqa employs a cross-lingual setting @xcite where relevant passages are retrieved in a high-resource language spoken in the corresponding region and answers are translated into the source language. to ensure utility of this dataset, we carefully select a relevant source 2 these questions are information-seeking in that they are written without seeing the answer, as is the case with real users of qa systems. we contrast this with the reading comprehension task where the question-writer sees the answer passage prior to writing the question; this genre of questions tends to have both higher lexical overlap with the question and elicit questions that may not be of broad interest. language (either english or french) based on its prevalence in the region corresponding to the query language. afriqa includes 12,000+ examples across 10 languages spoken in different parts of africa. the majority of the dataset's questions are centered around entities and topics that are closely linked to africa. this is an advantage over simply translating existing datasets into these languages. by building a dataset from the ground up that is specifically tailored to african languages and their corresponding cultures, we are able to ensure better contextual relevance and usefulness of this dataset. we conduct baseline experiments for each part of the open-retrieval qa pipeline using different translation systems, retrieval models, and multilingual reader models. we demonstrate that crosslingual retrieval still has a large deficit compared to automatic translation and retrieval; we also show that a hybrid approach of sparse and dense retrieval improves over either technique in isolation. we highlight interesting aspects of the data and discuss annotation challenges that may inform future annotation efforts for qa. overall, afriqa proves challenging for state-of-the-art qa models. we hope that afriqa encourages and enables the development and evaluation of more multilingual and equitable qa technology. the dataset will be released under the creative commons attribution 4.0 international (cc by 4.0) license. in summary, we make the following contributions: ## afriqa afriqa is a cross-lingual qa dataset that was created to promote the representation and coverage of under-resourced african languages in nlp research. we show examples of the data in ## tasks and baselines as part of the evaluation for afriqa, we follow the methodology proposed in @xcite and assess its performance on three different tasks: xor-retrieve, xor-pivotlanguagespan, and xor-full. each task poses unique challenges for cross-lingual information retrieval and qa due to the low-resource nature of many african languages. ## conclusion in this work, we take a step toward bridging the information gap between native speakers of many african languages and the vast amount of digital information available on the web by creating afriqa, the first open-retrieval cross-lingual qa dataset focused on african languages with 12,000+ questions. we anticipate that afriqa will help improve access to relevant information for speakers of african languages. by leveraging the power of cross-lingual qa, we hope to bridge the information gap and promote linguistic diversity and inclusivity in digital information access.