CLIR udit: Cross-Lingual Information Retrieval of Scientific Documents (2025)

Abstract: Cross-lingual information retrieval (CLIR) helps users find documents in languages different from their queries. This is especially important in academic search, where key research is often published in non-English languages. We present CLIRudit, a novel English-French academic retrieval dataset built from Érudit, a Canadian publishing platform. Using multilingual metadata, we pair English author-written keywords as queries with non-English abstracts as target documents, a method that can be applied to other languages and repositories. We benchmark various first-stage sparse and dense retrievers, with and without machine translation. We find that dense embeddings without translation perform nearly as well as systems using machine translation, that translating documents is generally more effective than translating queries, and that sparse retrievers with document translation remain competitive while offering greater efficiency. Along with releasing the first English-French academic retrieval dataset, we provide a reproducible benchmarking method to improve access to non-English scholarly content.

## introduction cross-lingual information retrieval (clir) helps users find documents written in languages different from their search queries. this removes the need for proficiency in multiple languages and makes it easier to access valuable information that might otherwise be missed because of language barriers. clir is especially important for academic research. while english is the main language for scientific communication, important work often exists in other languages, particularly in certain fields and historical contexts @xcite @xcite . researchers figure 1 : the clirudit dataset. we use articles with abstracts and keywords in both french and english. english keywords form the queries, with relevance judged by their presence in each article. documents consist of the french title, subtitle, and abstract. may overlook key work if they cannot search across languages, especially if they're unfamiliar with technical terms. english is also often used due to the expectation of finding more results, reinforcing bias against documents in other languages. modern information retrieval (ir) systems often use bi-encoder architectures for first-stage retrieval, separately encoding documents and queries as dense embeddings @xcite @xcite . multilingual extensions of these methods have been effective in general-domain clir @xcite @xcite @xcite . another common approach is to use machine translation (mt) to convert queries or documents to the same language before searching @xcite @xcite . technical texts often use specialized vocabulary and styles that present challenges for mt and multilingual embeddings @xcite . however, research on clir in technical domains is limited @xcite , and studies focusing specifically on academic content are even scarcer, typically relying on small, curated datasets @xcite . as a result, the effectiveness of clir methods for academic retrieval remains underexplored. we address this gap by introducing a dataset for cross-lingual academic search and benchmarking first-stage retrieval methods. our contributions are: our results show that dense embeddings without translation perform nearly as well as those using mt. document translation generally improves retrieval more than query translation. while sparse retrievers combined with document translation may not surpass the best dense multilingual methods, they remain competitive and offer advantages in search speed and indexing efficiency. ## related work this section reviews relevant research on academic clir, focusing on first-stage retrieval methods, datasets, and bilingual academic corpora. @xcite proposed a conceptual framework for clir, outlining three main strategies for firststage retrieval: document translation (dt), translating documents into the query language; query translation (qt), translating queries into the document language; and language-independent representations, encoding queries and documents into a shared vector space for direct retrieval. since we focus on single-stage retrieval, we do not address later steps of a retrieval pipeline, such as re-ranking or results fusion. ## models and methods this section describes the retrieval and mt methods, and evaluation metrics used for benchmarking. ## results and analysis this section analyzes the performance of retrieval and translation models on clirudit ( we now discuss key findings from the results. 1. without translation, dense retrievers excel, even without multilingual retrieval fine-tuning. nv-embed-v2 and plaid-x achieved the highest map, while bge-m-gemma2 led in recall@100. interestingly, nv-embed-v2 is not reported to have multilingual capabilities; though its fine-tuning data, which is english-only for retrieval, includes french in sts17 and sts22 sentence pairs @xcite . bm25 with the french analyzer performed poorly without mt due to the query-document language mismatch, but still had non-zero results. this shows clirudit has some query-document lexical overlap; manual inspection revealed shared terms like proper nouns, latin terms, and acronyms. among sparse models, splade++ outperformed mgte-sparse and bge-m3-sparse, likely thanks to query expansion mitigating the language mismatch. psq addresses this mismatch via probabilistic translation, reaching map comparable to larger dense models like me5 and mgte-dense. manual review showed qt can reduce recall by mistranslating proper nouns with identical crosslanguage spelling. for example, "goose bay" (a canadian town) was incorrectly translated as "baie aux oies" instead of remaining unchanged. ## discussion dense single-vector retrievers based on large decoder-only models (e.g., nv-embed-v2, bge-m-gemma2) achieve near gold translation-augmented performance without additional training, which may result from pretraining on large corpora and cross-lingual transfer capabilities. a smaller, clir-specialized model, plaid-x, also performed competitively; at the expense of needing language-and task-specific training data and having higher search latency due to its multi-vector design @xcite . both dense approaches avoid the overhead of translating the entire corpus, but large models may incur high index0.4 0.6 0.8 law philosophy literary studies humanities and social sciences social work sociology anthropology and ethnology urban studies geography overall semiotics education water and environment history language studies health sciences arts political science demography biology industrial relations theology women studies psychology management economics map figure 5: map of retrievers across clirudit disciplines. each dot represents a method's map in a discipline's queries, using its best translation method (excluding gold). dot colors indicate retrievers: croissant (pink) often performs worst, while the best varies by discipline. our dataset uses keyword-based queries, reflecting how authors describe their work to make it discoverable. this assumes users know the right keywords, shifting the dataset challenge to language differences in a technical domain rather than query formulation. this allows meaningful analysis, though it's unclear how system rankings might change with other types of queries, e.g., natural language questions. our approach aligns with other datasets using non-natural or generated queries, such as scidocs, dbpedia @xcite , wikiclir @xcite , and clirmatrix @xcite . like all ir datasets, ours has limitations in scope and collection method, so we encourage evaluation on many, diverse datasets. as the first english-french academic retrieval dataset, clirudit adds to this diversity and complements existing resources. ## conclusions we introduced a method for building clir datasets from bilingual metadata in scientific publications. by using keywords as queries and abstracts as documents, this approach enables automated, scalable creation of large evaluation resources without manual annotation or complex heuristics. we applied it to produce clirudit, the first english-french clir dataset for academic search, based on a realworld database. evaluations of single-stage methods on clirudit showed that: (1) state-of-the-art dense biencoders achieved strong cross-lingual performance without translation, nearing monolingual retrieval with gold translations; (2) sparse retrievers with document translation were competitive; and these results have practical implications for academic search systems. large dense retrievers deliver the best performance, but the strong results of sparse retrievers with document translation suggest a viable alternative that may be more practical to implement at scale. this is particularly relevant for academic publishing platforms like érudit that aim to make their content more discoverable to researchers. our method can be applied to other academic databases and language pairs, supporting broader research in cross-lingual access to scientific knowledge.