Mr. T y D i: A Multi-lingual Benchmark for Dense Retrieval (2021)

Abstract: We present Mr. TyDi, a multi-lingual benchmark dataset for mono-lingual retrieval in eleven typologically diverse languages, designed to evaluate ranking with learned dense representations. The goal of this resource is to spur research in dense retrieval techniques in non-English languages, motivated by recent observations that existing techniques for representation learning perform poorly when applied to out-of-distribution data. As a starting point, we provide zero-shot baselines for this new dataset based on a multi-lingual adaptation of DPR that we call “mDPR”. Experiments show that although the effectiveness of mDPR is much lower than BM25, dense representations nevertheless appear to provide valuable relevance signals, improving BM25 results in sparse–dense hybrids. In addition to analyses of our results, we also discuss future challenges and present a research agenda in multi-lingual dense retrieval. Mr. TyDi can be downloaded at https://github.com/castorini/mr.tydi .

## introduction retrieval approaches based on learned dense representations, typically derived from transformers, form an exciting new research direction that has received much attention of late. these dense retrieval techniques generally adopt a supervised approach to representation learning, where a labeled dataset is used to train two encoders-one for the queries and the other for texts from the corpus to be retrieved-whose output representation vectors are then compared with a simple comparison function such as inner product. retrieval against a large text corpus is typically formulated as nearest neighbor search and efficiently executed using offthe-shelf libraries. in the literature, this is known as a "bi-encoder" design. well-known examples include dpr @xcite , ance @xcite , and colbert @xcite , but there is much recent work along these lines @xcite @xcite , just to list a few papers. like all methods based on supervised machine learning, the effectiveness of the trained models on "out of distribution" (ood) samples is an important issue since it concerns model robustness and generalizability. for dense retrieval, training data typically comprise (query, relevant passage) pairs, and in this context, ood could mean that (1) the passage encoder is fed text from a different domain, genre, register, etc. than the training data, (2) the query encoder is fed queries that are different from the training queries, (3) the relationship between the inputs at inference time is different from the training samples (e.g., task variations), or (4) a combination of all of the above. it is, in fact, already known that dense retrieval techniques generalize poorly across different corpora, queries, tasks, etc. recently, @xcite constructed a benchmark to specifically evaluate the zero-shot transfer capabilities of dense retrieval models by creating a framework that unifies over a dozen retrieval datasets spanning diverse domains. in a zero-shot setting, the authors found that bm25 remained the most effective overall. that is, dense retrieval techniques trained on one dataset can spectacularly fail on another datasetexactly the out-of-distribution challenges we discussed above. in contrast, bm25, "just works" regardless of the corpus and queries, even though on "in distribution" samples, dense retrieval models are unequivocally more effective. thus, learned dense representations are not as general and robust as bm25 representations. this paper focuses on another aspect of the generalizability and robustness of learned representations for ranking: what if the encoders are applied to texts in languages different from the one they are trained in? our focus is on mono-ఐఆ#ఎ%ఎ&ఎ&-1)ీ ఉపగ/ 0 బర3వ5 ఎంత? (what is the weight of the irnss-1c satellite?) సమ#ద% ంల( *వ,ి ం.ే అ1 23 ద4 జంత7వ8 ఏ:; ? we view this work as having three main contributions: first, we construct and share mr. tydi, a multi-lingual benchmark dataset for mono-lingual retrieval in eleven diverse languages, designed to evaluate ranking with learned dense representations. this dataset can be viewed as the "openretrieval" condition of the tydi multi-lingual question answering (qa) dataset @xcite , and "mr" in mr. tydi stands for "multi-lingual retrieval". we describe the construction of this dataset and how it is different from existing resources. second, we report zero-shot baselines for mr. tydi, including a dense retrieval method based on a multi-lingual version of dpr @xcite ) that we call "mdpr". third, we present a number of initial findings about baseline results that highlight future challenges and begin to define a research agenda in multi-lingual dense retrieval. most interestingly, we find that although the zero-shot effectiveness of mdpr is much worse than bm25, dense representations appear to provide valuable relevance signals, improving bm25 results in sparse-dense hybrids. ## background and related work in presenting a new benchmark dataset, one important question to answer is: why is a new resource needed? we begin by addressing this question. the introduction already lays out the intellectual motivation for our work. thus, we focus here on explaining why existing datasets are not sufficient. the answer is summarized in figure 1 . mr. tydi is constructed from tydi @xcite , a question answering dataset covering eleven typologically diverse languages. for each language, the creators provided annotators with a prompt (the first 100 characters of a wikipedia article), who were asked to write a question that cannot answered by the snippet. then, for each question, annotators were given the top wikipedia article returned by google search and asked to label the relevance of each passage in the article as well as to identify a minimal answer span (if possible). given this procedure, the answer to the question may or may not be found in the passages from the selected article. the answer passages are always in the same language as the questions. note that the questions in different languages are not comparable as they are created independently rather than through translation. the weakness of tydi from our perspective is that it is essentially a machine reading comprehension task like squad @xcite because candidate passages are all included as part of the dataset (i.e., the passages are from the top wikipedia article returned by google search). instead, we need a resource akin to what qa researchers call the "open-domain" or "open-retrieval" task, where the problem involves retrieval from a much larger corpus (e.g., all of wikipedia) @xcite . thus, at a high level, mr. tydi can be viewed as an open-retrieval extension to tydi. @xcite created xor-tydi, a crosslingual qa dataset built on tydi by annotating answers in english wikipedia for questions tydi considered unanswerable in the original source (non-english) language. this was accomplished by randomly sampling 5,000 unanswerable (non-english) questions from tydi, and then searching english wikipedia articles for answers. specifically, each non-english question was first translated into english; then, annotators were given the top-ranked english wikipedia articles and asked to label passages containing the answer. the xor-tydi dataset contains three overlapping tasks, but all of them are focused on the cross-lingual condition. among the three tasks (xor-englishspan, xor-retrieve, and xor-full), xor-retrieve is most comparable to our work, but the retrieval target for the task is explicitly english wikipedia articles rather than wikipedia articles in the question's language. while the xor-full task requires systems to select answer spans from both english and targetlanguage wikipedia articles, the dataset does not provide ground truth for the intermediate retrieval step, thus it cannot be used for our evaluations. the annotations of xor-tydi do not allow us to examine mono-lingual retrieval in non-english languages because the creators started with "unanswerable" non-english tydi questions. furthermore, since all answer passage annotations were performed on english wikipedia, this doesn't help if we are interested in, for example, searching finnish wikipedia with finnish questions. another point of comparison worth discussing is mkqa @xcite , which comprises 10k question-answer pairs aligned across 26 typologically diverse languages. questions are paired with exact answers in the different languages, and evaluation is conducted in the open-retrieval setting by matching those answers in retrieved text. there are two main differences between our work and mkqa: first, mkqa was created via translation in order to achieve cross-lingual alignment. in addition to the possible translation artifacts that such a process might introduce, which @xcite discussed at length, we argue that forced alignment creates non-natural questions, for the simple reason that speakers of different languages are likely to be interested in different topics. this is different from the "geographically dependent" questions that mkqa tries to avoid. take the question "who starred in the movie bridge over the river kwai" as an example: while it does not involve any geographical preference, the question is probably less likely to be asked in, say, swahili, compared to in english or thai. second, the builders of mkqa explicitly made the decision to create "retrievalindependent answer annotations" that are linked to wikidata entities and a few other value types. this decision, we feel, restricts the range of natural language questions that are covered. the crosslingual aspect of the dataset appears to be primarily limited to entity translations, which likely do not cover a wide range of linguistic phenomena (which is the reason that we are interested in typologically diverse languages to begin with). thus, we believe that mr. tydi fills a gap in the evaluation space that is currently not occupied. many multi-lingual (both mono-lingual and cross-lingual) information retrieval and question answering datasets have been constructed over the past decades, via community-wide evaluations at trec, fire, clef, and nctir. these test collections are typically built on newswire articles, although some evaluations use wikipedia and scientific texts. while no doubt useful for evaluation, these test collections usually comprise only a small number of queries (at most a few dozen) with relevance judgments, which are insufficient to finetune dense retrieval models. furthermore, whereas tydi at least draws from comparable corpora (i.e., wikipedia articles), these test collections are built on corpora from much more diverse sources. this makes it difficult to generalize across different languages. for these reasons, the above-mentioned ir and qa test collections are not suitable for tackling the research questions we are interested in. corpus the formulation of any text ranking problem begins with a corpus @xmath0. as the starting point, we used exactly the same raw wikipedia dumps as tydi. relevance annotations in tydi are provided at the passage level (in the passage selection task), and thus we kept the same level of granularity in our corpus preparation. for articles covered by tydi (identified by the article titles), we retained the original passages. for articles that are not covered by tydi, we prepared passages using wiki-extractor foot_0 based on natural discourse units (e.g., two consecutive newlines in the wiki markup). unfortunately, @xcite did not precisely document their passage segmentation method, but based on manual examination of our results, the generated passages appear qualitatively similar to the tydi passages. the result of the corpus preparation process is, for each language, a collection of passages from the wikipedia articles in that language. to form the final passages that comprise the basic unit of retrieval, we prepend the title of the wikipedia article to each passage. this creates retrieval units that can be more readily understood in isolation. 2 task while mr. tydi is adapted from a qa dataset, our task is mono-lingual ad hoc retrieval. that is, given a question in language l, the task is to retrieve a ranked list of passages from c l , the wikipedia collection in the same language (prepared in the manner described above), where the retrieved passages are ranked according to their relevance to the given question. our assumption here is a standard "retriever-reader" framework @xcite or a multistage ranking architecture @xcite ), where we focus on the retriever (what ir researchers call candidate generation or first-stage retrieval). for end-to-end question answering, the output of the retriever would be fed to a reader for answer extraction. this focus on retrieval allows us to explore the research questions outlined in the introduction, and this formulation is consistent with previous work in dense retrieval, e.g., @xcite . questions and judgments to prepare the questions, we started with all questions provided by tydi and removed those without any answer passages or whose answer passages are all empty. we consider all non-empty annotated answer passages from tydi as relevant to the corresponding question in mr. tydi. we adopt the development set of tydi as our test set, since the original test data are not public. a new development set was created by randomly sampling 20% of questions from the original training set. we observed that some of the questions in tydi are shared between the training and development set (but labeled with different answer passages). in these cases, we retained the duplicate questions only in the training set. descriptive statistics for mr. tydi are shown in in other words, our judgments are far from exhaustive. this might be a cause for concern, but is a generally accepted practice in ir research due to the challenges of gathering complete judgments. the widely used ms marco datasets @xcite , for example, share this characteristic of having "sparse judgments". no claim is made about the exhaustiveness of the annotations, as both mr. tydi and ms marco provide only about one good answer per question. from a methodological perspective, findings based on ms marco "sparse judgments" are largely consistent with results from more expensive evaluation efforts (to gather more complete judgments), such as the trec deep learning tracks @xcite . we expect a similar parallel here: more exhaustive judgments will change the absolute scores, but will likely not affect the findings qualitatively. metrics we evaluate results in terms of reciprocal rank and recall at a depth k of 100 hits. the first metric quantifies the ability of a model to generate a good ranking, while the second metric provides an upper bound on end-to-end effectiveness (e.g., when retrieval results are fed to a reader for answer extraction). the setting of @xmath1. ## baselines we provide a few "obvious" baselines for mr. tydi as a starting point for future research: bm25 we report results with bag-of-words bm25, a strong traditional ir baseline, with the implementation provided by pyserini @xcite , which is built on the open-source lucene search library. lucene provides language-specific analyzers for nine of the eleven languages in mr. tydi; for these languages, we used the lucene implementations. for telugu (te) and swahili (sw), since lucene does not provide any language-specific implementations, we simply used its whitespace analyzer. we report bm25 scores on two conditions, with default and tuned k 1 and b parameters; the default settings are k 1 = 0.9, @xmath2.4. tuning was performed on the development set, on a per-language basis, via grid search on k 1 ∈ [0.1, 1.6] and @xmath3.1, 1.0], with step size 0.1, optimizing mrr@100. our retrieval experiments with mdpr can be characterized as zero shot: we applied the same mbert document encoder to convert passages from all eleven languages into dense vectors; similarly, we applied the same mbert question encoder to all questions. retrieval in each language was performed using facebook's faiss library for nearest neighbor search @xcite ; we used the flatip indexes. experiments were conducted using the same codebase as the dpr replication experiments of @xcite , with the pyserini toolkit @xcite . our choice of zero-shot mdpr as a baseline deserves some discussion. at a high level, we are interested in the generalizability of dense retrieval techniques in out-of-distribution settings (in this case, primarily different languages). operationally, our experimental setup captures the scenario where the model does not benefit from any exposure to the target task, even (question, relevant passage) pairs in the english portion of mr. tydi. this makes the comparison "fair" to bm25, which is similarly not provided any labeled data from the target task (in the case with default parameters). sparse-dense hybrid our hybrid technique combines the scores of sparse (bm25) and dense (mdpr) retrieval results. the final fusion score of each document is calculated by s sparse + α • s dense , where s sparse and s dense represent the scores from sparse and dense retrieval, respectively. this strategy is similar to the one described by @xcite . we take 1000 hits from mdpr and 1000 hits from bm25 and normalize the scores from each into [0, 1] since the range of the two types of scores otherwise are quite different. if one hit isn't found in the other, the normalized score for that hit is set to zero. the weight α was tuned in [0, 1] with a simple line search on the development set by optimizing mrr@100 with step size 0.01. ## results and analysis we performed experiments on mr. tydi v1.1, where each passage contains the title of the wikipedia article and the passage text. ## future work mr. tydi provides a resource to begin exploring mono-lingual ad hoc retrieval with both dense and sparse retrieval techniques. in this paper, we have focused primarily on zero-shot baselines. although zero-shot dense retrieval (mdpr) does not appear to be effective by itself, relevance signals from the model do appear to be complementary to sparse retrieval (bag-of-words bm25). we have identified how they are complementary (better recall vs. better ranking), but the behavior varies across languages, and we do not yet have an explanation for why; for example, do the typological characteristics of the language play a role? for our experiments, we have decided to focus on zero-shot effectiveness because it serves as the natural baseline of any technique that tries more sophisticated approaches. thus, the baselines here are foundational to any future work. we have explicitly decided not to report any language-specific fine-tuning results here, although preliminary experiments suggest that such techniques do bring about benefits. we have not yet systematically explored the broad design space of what @xcite calls "multi-step fine-tuning strategies", paralleling the explorations of @xcite in the context of transformer-based reranking models. there are many possible variations, for example, how many languages to use, what order to sequence data from different languages, possible data augmentation using machine translation, complementary data from other tasks, etc. there are a number of experiments that will allow us to tease apart the effects of language versus other aspects of training data distribution (e.g., nq vs. tydi). exploration of this vast design space is the focus of our immediate future work. in addition, we believe that our dataset can provide a probe to examine the nature of multi-lingual transformer models. our experimental results show that absolute effectiveness varies quite a bit across languages. some of these variations may be due to the nature of the queries, the size of the corpora, etc. however, we hypothesize that inherent properties of the transformer model play important roles as well, e.g., the size of the pretraining corpus in each language, typological and other innate characteristics of the languages, etc. we hope that mr. tydi can help us untangle some of these issues. ## conclusion in this work, we introduce mr. tydi, a multilingual benchmark dataset for mono-lingual retrieval in eleven typologically diverse languages, built on tydi. we describe zero-shot experiments using bm25, mdpr, and a sparse-dense hybrid. the experimental results are not surprising: as is already known from complementary experiments, dense retrieval techniques do not generalize well to out-of-distribution input. however, we find that even poor dense retrieval results provide valuable relevance signals in a sparse-dense hybrid. of course, this is only the starting point. with mr. tydi, we now have a resource to explore our motivating research questions regarding the behavior of dense retrieval models when fed "out of distribution" data, and from there, devise techniques to increase the robustness and generalizability of our techniques. the potential broader impact of this work is more equitable distribution of information access capabilities across diverse languages of the world-to help non-english speakers access relevant information in their own languages.