CLIRM atrix: A massively large collection of bilingual and multilingual datasets for Cross-Lingual Information Retrieval (2020)

Abstract: We present CLIRMatrix, a massively large collection of bilingual and multilingual datasets for Cross-Lingual Information Retrieval extracted automatically from Wikipedia. CLIRMatrix comprises (1) BI-139, a bilingual dataset of queries in one language matched with relevant documents in another language for 139x138=19,182 language pairs, and (2) MULTI-8, a multilingual dataset of queries and documents jointly aligned in 8 different languages. In total, we mined 49 million unique queries and 34 billion (query, document, label) triplets, making it the largest and most comprehensive CLIR dataset to date. This collection is intended to support research in end-to-end neural information retrieval and is publicly available at [url]. We provide baseline neural model results on BI-139, and evaluate MULTI-8 in both single-language retrieval and mix-language retrieval settings.

## introduction cross-lingual information retrieval (clir) is a retrieval task in which search queries and candidate documents are written in different languages. clir can be very useful in some scenarios. for example, a reporter may want to search foreignlanguage news to obtain different perspectives for her story; an inventor may explore the patents in another country to understand prior art. traditionally, translation-based approaches are commonly used to tackle the clir task @xcite @xcite : the query translation approach translates the query into the same language of the documents, whereas the document translation approach translates the document into the same language as the query. both approaches rely on a machine translation (mt) system or bilingual dictionary to map queries and documents to the same language, then employ a monolingual information retrieval (ir) engine to find relevant documents. recently, the research community has been actively looking at end-to-end solutions that tackle the clir task without the need to build mt systems. this line of work builds upon recent advances in neural information retrieval in the monolingual setting, c.f. @xcite . there are proposals to directly train end-to-end neural retrieval models on clir datasets @xcite or mt bitext @xcite . one can also exploit cross-lingual word embeddings to train a clir model on disjoint monolingual corpora @xcite . despite the growing interest in end-to-end clir, the lack of a large-scale, easily-accessible clir dataset covering many language directions in high-, mid-and low-resource settings has detrimentally affected the clir community's capability to replicate and compare with previously published work. for example, among the widely-used datasets, the clef collection @xcite covers many languages but is not large enough for training neural models. the more recent iarpa mate-rial/openclir collection @xcite , is not yet publicly accessible. this motivates us to design and build clirmatrix, a massively large collection of bilingual and multilingual datasets for clir. we construct clirmatrix from wikipedia in an automated manner, exploiting its large variety of languages and massive number of documents. the core idea is to synthesize relevance labels via an existing monolingual ir system, then propagate the labels via wikidata links that connect documents in different languages. in total, we were able to mine 49 million unique queries in 139 languages and 34 billion (query, document, label) triplets, creating a clir collection across a matrix of 139 × 138 = 19, 182 language pairs. from this raw collection, we introduce two datasets: see figure 1 for a comparison of bi-139 and multi-8. the former facilitates the evaluation of bilingual retrieval over a wide variety of languages, while the latter supports research in mixed-language retrieval (a.k.a multilingual retrieval @xcite ), which is an interesting yet relatively under-explored problem. for both, the train sets are large enough to enable the training of the neural ir models. we hope clirmatrix is useful and can empower further developments in this field of research. to summarize, our contributions are: clirmatrix is publicly available at https:// github.com/ssun32/clirmatrix. for the english query "barack obama", first a monolingual ir engine (elasticsearch) labels documents in english; then wikidata links are exploited to propagate the label to the corresponding chinese documents, which are assumed to be topically similar. ## methodology let q x be a query in language x, and d y be a document in language y. a bilingual clir dataset consists of i triples suppose there are j documents in total. in the full collection search setup, the index j ranges from 1, . . . , j, meaning that each query q x i searches over the full set of documents {d y ij } @xmath0,...,j . in the re-ranking setup, each query q x i searches over a subset of documents obtained by an initial fullcollection retrieval engine: {d y ij } @xmath1,...,k i , where k i j. for practical reasons, machine learning approaches to ir focus on the re-ranking setup with k i set to 10∼1000 @xcite . we follow the re-ranking setup here. we now describe the main intuition of our construction method and detail various components and design choices in our pipeline. ## related work information retrieval (ir) has made a tremendous amount of progress, shifting focus from traditional bag-of-world retrieval functions such as tfidf @xcite ) and bm25 @xcite , to neural ir models @xcite @xcite which have shown promising results on multiple monolingual ir datasets. recent advances in pretrained language models such as bert @xcite have also led to significant improve-ments in ir tasks. for example, macavaney et al. cross-lingual information retrieval (clir) is a sub-field of ir that is becoming increasingly important as new documents in different languages are being generated every day. the field has progressed from translation-based methods @xcite @xcite to recent neural clir models @xcite @xcite ) that rely on cross-lingual word embeddings. in contrast to the wide availability of monolingual ir datasets @xcite , cross-lingual and multilingual ir dataset #lang manual? multilingual? #query #document #triplets (clef 2000-2003) 10 yes yes 2.2k 1.1m 33k (material, 2017) 7 yes no 11.5k 90k ∼20k (schamoni et al., 2014) 2 no no 245k 1.2m 3.2m (sasaki et al., 2018) 25 no no 10.9m 23.9m 40.1m clirmatrix bi-139 raw 139 no no 49.3m 50.5m 34.1b clirmatrix bi-139 base 139 no no 27.5m 50.1m 22.3b clirmatrix multi-8 8 no yes 10.4k 13.4m 72.8m extracting clir datasets from wikipedia has been explored in previous work. @xcite build a german-english bilingual clir dataset from wikipedia, which contains 245,294 german queries and 1,226,741 english documents. they convert the first sentences from german wikipedia documents into queries and follow wikipedia's interlanguage links to find relevant documents in english. @xcite apply the same techniques and release a larger clir dataset which contains english queries and relevant documents in 25 languages. both datasets truncate the documents to the first 200 tokens and rely on bidirectional inter-article links to find partially relevant documents. our contribution differs in three important aspects: (i) bi-139 is a significantly larger dataset, covering more languages and more documents. (ii) multi-8 provides a new multilingual retrieval setup, not previously available. (iii) we argue that our method can reliably find more relevant documents by propagating search results from monolingual ir systems to other languages via wikidata. this is in contrast to directly using bidirectional links extracted from wikipedia documents to determine relevance, which are much sparser. further, our method allows for more finergrained levels of relevance (e.g. as opposed to binary relevance), making the dataset more challenging. a comparison of various existing clir datasets is presented in ## conclusion and future work we present clirmatrix, the largest and the most comprehensive collection of bilingual and multilingual clir datasets to date. the bi-139 dataset supports clir in 139×138 language pairs, whereas the multi-8 dataset enables mix-language retrieval in 8 languages. the large number of supported language directions allows the research community to explore and build new models for many more languages, especially the low-resource ones. we document baseline ndcg results using a neural ranker based on multilingual bert. our mixlanguage retrieval experiments on multi-8 show that a single multilingual model can significantly outperform the combination of multiple bilingual models. for future work, we think it will be interesting to look at: