MIRACL : A Multilingual Retrieval Dataset Covering 18 Diverse Languages (2023)

Abstract: MIRACL is a multilingual dataset for ad hoc retrieval across 18 languages that collectively encompass over three billion native speakers around the world. This resource is designed to support monolingual retrieval tasks, where the queries and the corpora are in the same language. In total, we have gathered over 726k high-quality relevance judgments for 78k queries over Wikipedia in these languages, where all annotations have been performed by native speakers hired by our team. MIRACL covers languages that are both typologically close as well as distant from 10 language families and 13 sub-families, associated with varying amounts of publicly available resources. Extensive automatic heuristic verification and manual assessments were performed during the annotation process to control data quality. In total, MIRACL represents an investment of around five person-years of human annotator effort. Our goal is to spur research on improving retrieval across a continuum of languages, thus enhancing information access capabilities for diverse populations around the world, particularly those that have traditionally been underserved. MIRACL is available at http://miracl.ai/.

## introduction information access is a fundamental human right. specifically, the universal declaration of human rights by the united nations articulates that ''everyone has the right to freedom of opinion and expression'', which includes the right ''to seek, receive, and impart information and ideas through any media and regardless of frontiers'' (article 19). information access capabilities such as search, question answering, summarization, and recommendation are important technologies for safeguarding these ideals. with the advent of deep learning in nlp, ir, and beyond, the importance of large datasets as drivers of progress is well understood @xcite . for retrieval models in english, the ms marco datasets @xcite @xcite have had a transformative impact in advancing the field. similarly, for question answering (qa), there exist many resources in english, such as squad @xcite , triviaqa @xcite , and natural questions @xcite . we have recently witnessed many efforts in building resources for non-english languages, for example, clirmatrix @xcite , xtreme @xcite , mkqa @xcite , @xcite , tydi qa @xcite , xor-tydi @xcite , and mr. tydi @xcite . these initiatives complement multilingual retrieval evaluations from trec, clef, ntcir, and fire that focus on specific language pairs. nevertheless, there remains a paucity of resources for languages beyond english. existing datasets are far from sufficient to fully develop information access capabilities for the 7000+ languages spoken on our planet @xcite . our goal is to take a small step towards addressing these issues. to stimulate further advances in multilingual retrieval, we have built the miracl dataset on top of mr. tydi @xcite , comprising human-annotated passage-level relevance judgments on wikipedia for 18 languages, totaling over 726k query-passage pairs for 78k queries. these languages are written using 11 distinct scripts, originate from 10 different language families, and collectively encompass more than three billion native speakers around the world. they include what would typically be characterized as high-resource languages as well as low-resource languages. figure 1 shows a sample thai query with a relevant and a non-relevant passage. in total, the miracl dataset represents over 10k hours, or about five person-years, of annotator effort. along with the dataset, our broader efforts included organizing a competition at the wsdm 2023 conference that provided a common evaluation methodology, a leaderboard, and a venue for a competition-style event with prizes. to provide starting points that the community can rapidly build on, we also share reproducible baselines in the pyserini ir toolkit @xcite . compared to existing datasets, miracl provides more thorough and robust annotations and broader coverage of the languages, which include both typologically diverse and similar language pairs. we believe that miracl can serve as a high-quality training and evaluation dataset for the community, advance retrieval effectiveness in diverse languages, and answer interesting scientific questions about cross-lingual transfer in the multilingual retrieval context. ## background and related work the focus of this work is the standard ad hoc retrieval task in information retrieval, where given a corpus c, the system's task is to return for a given query q an ordered list of top-k passages from c that maximizes some standard quality metric such as ndcg. a query q is a well-formed natural language question in some language l n and the passages draw from the same language c n . thus, our focus is monolingual retrieval across diverse languages, where the queries and the corpora are in the same language (e.g., thai queries searching thai passages), as opposed to crosslingual retrieval, where the queries and the corpora are in different languages (e.g., searching a swahili corpus with arabic queries). as mentioned in the introduction, there has been work over the years on building resources for retrieval in non-english languages. below, we provide a thorough comparison between miracl and these efforts, with an overview in ## dataset construction to build miracl, we hired native speakers as annotators to provide high-quality queries and relevance judgments. at a high level, our workflow comprised two phases: first, the annotators were asked to generate well-formed queries based on ''prompts'' @xcite (details in section 4.2). then, they were asked to assess the relevance of the top-k query-passage pairs produced by an ensemble baseline retrieval system. an important feature of miracl is that our dataset was not constructed via crowd-sourced workers, unlike other previous efforts such as squad @xcite . instead, we hired 31 annotators (both part-time and full-time) across all languages. each annotator was interviewed prior to being hired and was verified to be a native speaker of the language they were working in. our team created a consistent onboarding process that included carefully crafted training sessions. we began interviewing annotators in mid-april 2022; dataset construction began in late april 2022 and continued until the end of september 2022. throughout the annotation process, we constantly checked randomly sampled data to monitor annotation quality (see section 4.3). whenever issues were detected, we promptly communicated with the annotators to clarify the problems so that they were resolved expeditiously. constant interactions with the annotators helped minimize errors and misunderstandings. we believe that this design yielded higher quality annotations than what could have been obtained by crowd-sourcing. in total, miracl represents over 10k hours of assessor effort, or around five person-years. we offered annotators the hourly rate of @xmath011.50 usd/hr. ## miracl analysis to better characterize miracl, we present three separate analyses of the dataset we have built. ## conclusion in this work, we present miracl, a new high-quality multilingual retrieval dataset that represents approximately five person-years of annotation effort. we provide baselines and present initial explorations demonstrating the potential of miracl for studying interesting scientific questions. although the wsdm cup challenge associated with our efforts has ended, we continue to host a leaderboard to encourage continued participation from the community. while miracl represents a significant stride towards equitable information access, further efforts are necessary to accomplish this objective. one obvious future direction is to extend miracl to include more languages, especially low-resource ones. another possibility is to augment miracl with cross-lingual retrieval support. however, pursuing these avenues demands additional expenses and manual labor. nevertheless, miracl already provides a valuable resource to support numerous research directions: it offers a solid testbed for building and evaluating multilingual versions of dense retrieval models @xcite , late-interaction models @xcite , as well as reranking models @xcite , and will further accelerate progress in multilingual retrieval research @xcite @xcite . billions of speakers of languages that have received relatively little attention from researchers stand to benefit from improved information access.