A fri CLIRM atrix: Enabling Cross-Lingual Information Retrieval for A frican Languages (2022)

Abstract: Language diversity in NLP is critical in enabling the development of tools for a wide range of users.However, there are limited resources for building such tools for many languages, particularly those spoken in Africa.For search, most existing datasets feature few or no African languages, directly impacting researchers’ ability to build and improve information access capabilities in those languages.Motivated by this, we created AfriCLIRMatrix, a test collection for cross-lingual information retrieval research in 15 diverse African languages.In total, our dataset contains 6 million queries in English and 23 million relevance judgments automatically mined from Wikipedia inter-language links, covering many more African languages than any existing information retrieval test collection.In addition, we release BM25, dense retrieval, and sparse–dense hybrid baselines to provide a starting point for the development of future systems.We hope that these efforts can spur additional work in search for African languages.AfriCLIRMatrix can be downloaded at https://github.com/castorini/africlirmatrix.

## introduction the ever-increasing amounts of information on the web in different languages highlight the need for systems that enable users to search in one language and retrieve relevant documents in another. this search task, commonly known as cross-lingual information retrieval (clir), is becoming increasingly important. clir can break down language barriers between information seekers and the extensive collections of documents that are available in diverse languages. one common approach to clir takes advantage of machine translation and monolingual information retrieval @xcite . the documents and queries are translated into the same language before search occurs. this translation is often performed using a variety of sources, including parallel corpora, bilingual dictionaries, and machine translation (mt) systems. the effectiveness of this approach relies heavily on translation quality, which may be a bottleneck for low-resource languages where high-quality translations are not readily available. to address this challenge, researchers have recently explored the use of pretrained multilingual models @xcite . examples such as mbert @xcite and xlm-r @xcite are often pretrained on a large collection of multilingual texts, enabling the models to learn representations across different languages. the use of multilingual models for clir often builds on techniques that have previously been applied to monolingual retrieval @xcite . regardless of approach, modern neural-based clir models are data hungry, typically requiring large amounts of query-document pairs that have been annotated with relevance labels. such annotated data are expensive to obtain, especially for low-resource african language pairs. although there is ongoing research on training multilingual models for dense retrieval in low-resource settings @xcite , there are still not enough resources for these languages. existing clir datasets do contain some african languages, such as clirmatrix @xcite and the material corpora @xcite . however, these collections contain only a few african languages, a tiny fraction of the 2000+ languages spoken on the continent with hundreds of millions of speakers @xcite . the paucity of data hinders progress in developing information access capabilities for africa. as a small step towards plugging this gap, we introduce africlirmatrix, a new test collection for cross-lingual information retrieval containing geographically diverse african languages. this resource comprises english queries with querydocument relevance judgments in 15 african languages automatically mined from wikipedia. although we only cover a small set of languages, our resource already represents a substantial enhancement over existing datasets, as africlirmatrix covers geographically diverse languages that are collectively spoken by 340 million people in africa and worldwide. we hope that this resource will spur research in retrieval techniques and motivate the development of more robust datasets for information retrieval in african languages. as a start, we provide a number of baselines for researchers to build on: bm25, a multilingual adaptation of dpr known as "mdpr", and a hybrid approach combining the two. ## africlirmatrix africlirmatrix is a new information retrieval test collection comprising queries and documents in 15 diverse african languages mined from wikipedia, the largest such dataset that we are aware of. we focus on cross-lingual information retrieval with queries in english and documents in various african languages, listed in extraction pipeline: our mining pipeline is similar to the one used in @xcite . for every "source" wikipedia article in language l, there exist inter-language links that connect the source article to articles about the same topic in other languages. we leverage these connections to extract queries and a set of relevant articles in english, and then use wikidata backlinks to find relevant articles in other languages if they are available. we use english article titles as queries because they are readily available, span multiple domains, and have articles linked to more languages than any other language in wikipedia. however, our pipeline also supports other forms of queries, for example, @xcite used the first sentence in each article in their dataset. to find relevant articles, we use each query to retrieve a set of 100 articles in english using a bagof-words retrieval system (elasticsearch). 2 inter1 given that bm25 scores reflect how relevant a document (article) is to a given query, we use the scores to generate relevance judgments for the retrieved documents (articles). the scores are normalized and then converted into discrete relevance grades using the jenks natural break optimization algorithm @xcite . the documents are originally judged on a scale of 0 to 6, with 0 being irrelevant and 6 being the most relevant. a score of 0 is assigned to all documents not retrieved by the monolingual english pipeline using elasticsearch, while a score of 6 is assigned to documents from articles directly connected to the title queries. ## baselines as a starting point for future research, we release bm25, mdpr, and sparse-hybrid baselines for africlirmatrix. for each language, we split the extracted queries into training and test sets, as shown in ## conclusion and future work to spur interest in information retrieval research and development for african languages, we introduce a new dataset for cross-lingual information retrieval in 15 languages across different african regions. africlirmatrix is a collection of bilingual datasets with english queries and documents in 15 african languages. in addition to releasing the resource, we also provide baselines as a starting point for further research.