MINERS : Multilingual Language Models as Semantic Retrievers (2024)

Abstract: Words have been represented in a high-dimensional vector space that encodes their semantic similarities, enabling downstream applications such as retrieving synonyms, antonyms, and relevant contexts. However, despite recent advances in multilingual language models (LMs), the effectiveness of these models’ representations in semantic retrieval contexts has not been comprehensively explored. To fill this gap, this paper introduces the MINERS, a benchmark designed to evaluate the ability of multilingual LMs in semantic retrieval tasks, including bitext mining and classification via retrieval-augmented contexts. We create a comprehensive framework to assess the robustness of LMs in retrieving samples across over 200 diverse languages, including extremely low-resource languages in challenging cross-lingual and code-switching settings. Our results demonstrate that by solely retrieving semantically similar embeddings yields performance competitive with state-of-the-art approaches, without requiring any fine-tuning.

## introduction language models (lms) play a crucial role in learning natural language representations @xcite @xcite @xcite and have been successfully applied to various natural language processing (nlp) tasks, such as document retrieval @xcite . existing benchmarks have systematically evaluated lms to provide empirical assessments of their performance across a range of embedding tasks. some notable benchmarks include big-bench @xcite , mteb @xcite , semeval @xcite , and beir benchmark @xcite . mteb, in particular, has been established as a comprehensive benchmark for evaluating the * the work was conducted outside capital one. effectiveness of embeddings in downstream nlp applications. however, their analysis of the multilingual space has been limited to bitext mining, without further exploration of how these embeddings can be utilized in other multilingual downstream tasks. the advancement of multilingual lms is remarkable, demonstrating impressive capabilities in adapting to new languages through finetuning @xcite , learning from few-shot samples via incontext learning (icl) @xcite @xcite @xcite , enabling crosslingual zero-shot transfer @xcite , and incorporating language-specific adapters @xcite . this exploration now includes low-resource and regional languages not part of the pretraining phase, promoting nlp research for underrepresented languages @xcite @xcite . however, multilingual lms face two key challenges: (1) the lack of a comprehensive benchmark for evaluating effectiveness in semantic retrieval, and (2) limited understanding of code-switching (cs) texts common in multilingual communities. current cs evaluations focus on model finetuning benchmarks @xcite @xcite , without deeply exploring their potential as multilingual retrievers. recent studies by @xcite have primarily focused on semantic similarity using encoder lms in zero-shot crosslingual settings but have not explored their application in generative lms. this gap presents an opportunity to leverage these models as context providers for multilingual generative lms @xcite . in this paper, we introduce miners, the first benchmark designed to assess the multilingual lms' ability in semantic retrieval across various tasks. miners evaluates the representation of dense vectors in multiple tasks, including bitext retrieval, retrieval-based classification, and icl classification. we have developed miners to be a reproducible and reliable benchmark that utilizes high-dimensional multilingual vector representations. notably, these tasks do not require any fine-tuning. the paper's contribution can be summarized as follows: ## conclusion this paper introduces miners, a benchmark for evaluating the efficacy of multilingual lms in semantic retrieval tasks, including bitext retrieval and classification through semantic search and retrievalaugmented contexts. our framework rigorously assesses lms' robustness in retrieving samples from over 200 languages. empirical results demonstrate that our method, which focuses on retrieving semantically similar vector representations, achieves performance comparable to state-of-the-art finetuned approaches, without requiring fine-tuning across multiple datasets and languages. we also explore the mechanisms behind these representations, offering insights to improve the efficiency and accuracy of label retrieval methods. our research aims to pave the way for future exploration and optimization in semantic retrieval and classification, ultimately contributing to more robust and adaptable nlp systems.