M u S e CLIR : A Multiple Senses and Cross-lingual Information Retrieval Dataset (2022)

Abstract: This paper addresses a deficiency in existing cross-lingual information retrieval (CLIR) datasets and provides a robust evaluation of CLIR systems’ disambiguation ability. CLIR is commonly tackled by combining translation and traditional IR. Due to translation ambiguity, the problem of ambiguity is worse in CLIR than in monolingual IR. But existing auto-generated CLIR datasets are dominated by searches for named entity mentions, which does not provide a good measure for disambiguation performance, as named entity mentions can often be transliterated across languages and tend not to have multiple translations. Therefore, we introduce a new evaluation dataset (MuSeCLIR) to address this inadequacy. The dataset focusses on polysemous common nouns with multiple possible translations. MuSeCLIR is constructed from multilingual Wikipedia and supports searches on documents written in European (French, German, Italian) and Asian (Chinese, Japanese) languages. We provide baseline statistical and neural model results on MuSeCLIR which show that MuSeCLIR has a higher requirement on the ability of systems to disambiguate query terms.

## introduction cross-lingual information retrieval (clir) is a subfield of information retrieval (ir) where the task is to retrieve documents in language y using queries in language x. frequently, it is a combined process of translation and conventional ir. for efficiency, it is more common to translate queries from language x to language y than to translate documents from language y to language x. lexical ambiguity is a problem, and disambiguation found beneficial, in many natural language processing tasks, including machine translation @xcite , information extraction @xcite and information retrieval @xcite . this problem is english query translated query (a) karen carroll (judge) karencarroll法官 (b) brian duffy (chef) brianduffy主廚 (c) larry andersen larryandersen (d) el cacao, veraguas elcacaoveraguas despite its importance, translation ambiguity problem has received relatively little attention by clir researchers, and has been overlooked by existing clir datasets. many applications of clir, and thus many existing datasets, are dominated by searches for named entity mentions. for example, in the bi-139 english-chinese dataset @xcite , about 73% of the queries contain at least one named entity. these do not tend to have multiple translations and, as illustrated by by the query examples from bi-139 shown in in the real world, the need to search for unnamed entity mentions exists. for example, on a global topic such as conservation @xcite , researchers might want to collect information about bats from around the globe. bat is not a named entity and is named differently in different languages when referred to as an animal (italian: chiroptera; chinese: 蝙蝠). hence, retrieval systems need to distinguish between the translation of bat as an animal and bat as a piece of stick-like equipment in sports, based on the context information. however, due to a large amount of unambiguous named entity mentions, existing clir datasets are not adequate to train and evaluate the disambiguation ability of systems. we introduce museclir, a new and a more fine-grained evaluation dataset. museclir is designed to specifically evaluate systems' ability to carry out disambiguation in clir. it is derived from wikipedia, a free and open-sourced resource. wikipedia contains many pages that exist in multiple languages and thus makes wikipedia a good resource for clir @xcite @xcite . assuming every possible translation represents a word sense, common nouns with more than one possible translation are chosen. by doing so, we can minimise the number of named entity mentions appearing in queries, and test whether systems are able to rank documents more highly which contain the correct translation (in context) over other possible translations of the ambiguous words in queries. we introduce museclir, a new evaluation dataset that assesses the ability of systems to disambiguate ambiguous query terms. museclir supports searches on documents written in european (french, german, italian) and asian (chinese, japanese) languages. our codes are available on github 1 . users can reproduce and extend museclir to other languages. in section 3, we provide the construction method of museclir and the statistics. in section 4, museclir is used as a benchmark to evaluate existing clir systems: bm25 and multilingual bert (mbert) ranker @xcite . the results indicate that, given similar types of queries, existing clir systems perform more poorly on museclir compared to other existing datasets, showing their inadequacy and the need of museclir to determine the most appropriate system in the real-world scenario. elasticsearch foot_0 is an open-source search engine that implements the bm25 algorithm @xcite and has built-in analysers that handle tokenisation and stemming. here, we employ elasticsearch 6.5.4 with default parameters foot_1 ; smartcn and kuromoji analyser are used when handling the chinese and japanese documents, respectively. neural approach recently, end-to-end clir models have attracted more attention. these systems align queries and documents into the same space and perform matching in this aligned space. large pre-traine d language models (plm), such as bert @xcite , is commonly adopted as the encoder @xcite @xcite and impressive results have been achieved. as clir involves multiple languages, clir systems usually utilise multilingual language models, like mbert, to map queries and documents into a shared space, bypassing the translation step. the mbert ranker used here is a reimplementation of the vanilla bert ranker proposed by @xcite . the vanilla bert ranker adopted the fine-tuning paradigm with a linear layer stacked on top of bert @xcite . following sun and duh (2020), the encoder is replaced with a pre-trained mbert mode foot_2 . the [cls] embedding at the final layer of mbert that represents the query-document pair is used. when training, the positive sample is a querydocument pair with relevance labels larger than 0; negative otherwise. the network is trained to optimise pairwise hinge loss with adam optimiser and updates the weights in the last linear layer. datasets museclir is compared against two existing datasets, both also developed from wikipedia: bi-139 from clirmatrix @xcite and wikiclir dataset introduced by @xcite . clirmatrix contains two subsets: bi-139 (bilingual dataset) and multi-8 (multilingual dataset 5 ). as this paper is not studying multilingual ir, bi-139 (base version) is considered. queries in wikiclir are the first sentences from the english wikipedia pages with page titles removed; the average query length is 20 tokens. as page titles are kept in museclir, we append page titles to their original queries forming wiki-clir title for fair comparisons. queries in bi-139 are wikipedia page titles of 3 tokens on average. in both sets, documents are the first 200 tokens of a page that contains the main gist of the topic. ## dataset construction museclir is an english-centric dataset where all queries are in english that makes use of wikipedi-aapi 6 . common nouns with multiple translations are chosen from muse @xcite bilingual dictionaries. wikipedia provides a disambiguation page for potentially ambiguous article titles. this page contains links to other possible subtopics in addition to the main topic (selected nouns in our case). for example, window can refer to architecture, rectangle display on computers, etc. this page will provide links to the corresponding articles. here, we assume each link within the disambiguation page is associated with a sense of the noun. in order to conserve the overall level of ambiguity, we remove nouns with less than 2 links on the disambiguation page. administration pages like help, disambiguation, etc. are then removed. before adding a linked page to the final collection, we also check that i) the page contains an inter-language link to the desired target language; ii) the noun exists in the page title; and iii) the summary of the page in both english and the target language is not empty. we initialise the construction based on common nouns to minimise the number of named entity mentions. this method is easily adapted to other parts of speech, but we focus on polysemous nouns here. ## evaluation two methods are considered in the following experiments: bm25, an unsupervised probabilistic approach, and an mbert ranker, a supervised neural network @xcite . typically, there are two ranking stages in ir systems. at the initial stage, each query will search over all documents and then rerank on a subset of documents returned from the first stage. however, the central challenge of museclir is a cross-lingual and cross-sense problem, not the conventional ir task. baseline models are two individual ranking models. bm25 ranks the complete document collection, and following @xcite , mbert ranker ranks 100 documents at test time. results are reported using map@10 (mean average precision), calculated using pytrec_eval 8 (van gysel and de rijke, 2018). as the metric considered will cut off at 10, we limit bm25 to return 10 documents. queries are translated using muse @xcite bilingual dictionaries per token, and the first possible translation is returned, disregarding other possible translations. out-of-vocabulary tokens will use the original form (i.e. english). ## conclusion and further work to address a deficiency in existing clir datasets, we introduce museclir, a clir dataset that has been designed to challenge the ability of models to deal with ambiguous query terms. this dataset focused on polysemous common nouns with more than one possible translation, and which are regarded as ambiguous on wikipedia. we argue that museclir is a more suitable evaluation dataset for clir than pre-existing datasets. our method is replicable and extendable to other language pairs and other parts of speech. in the future, we also intend to test models that are trained with museclir on real-world data and standard clir test sets.