The R ussian-focused embedders’ exploration: ru MTEB benchmark and R ussian embedding model design (2025)

Abstract: Embedding models play a crucial role in Natural Language Processing (NLP) by creating text embeddings used in various tasks such as information retrieval and assessing semantic text similarity. This paper focuses on research related to embedding models in the Russian language. It introduces a new Russian-focused embedding model called ru-en-RoSBERTa and the ruMTEB benchmark, the Russian version extending the Massive Text Embedding Benchmark (MTEB). Our benchmark includes seven categories of tasks, such as semantic textual similarity, text classification, reranking, and retrieval.The research also assesses a representative set of Russian and multilingual models on the proposed benchmark. The findings indicate that the new model achieves results that are on par with state-of-the-art models in Russian. We release the model ru-en-RoSBERTa, and the ruMTEB framework comes with open-source code, integration into the original framework and a public leaderboard.

## introduction text embeddings play an important role in many natural language processing (nlp) tasks, from clustering to semantic textual similarity (sts) and information retrieval @xcite . the community has addressed this demand by releasing several powerful text embedding models (or embedders) @xcite @xcite . however, there is still a lack of such embedders developed specifically for the russian language. the most popular russian-oriented models, such as rubert-tiny2 1 , sbert large-nlu-ru 2 , and sbert large-mt-nlu-ru 3 , have been released several years ago and thus do 1 https://huggingface.co/cointegrated/ rubert-tiny2 2 https://huggingface.co/ai-forever/sbert_ large_nlu_ru 3 https://huggingface.co/ai-forever/sbert_ large_mt_nlu_ru not include modern data in their training corpora. the latest models are based on an outdated version of the rubert @xcite foot_0 model as a backbone. moreover, being monolingual, they can not profit from knowledge transfer between languages. given the usability of such text embeddings, evaluating their quality and the corresponding embedders is also important. one general approach is to evaluate text embeddings on a set of standard text embedding tasks (classification, clustering, etc.) for english, massive text embedding benchmark, or mteb @xcite , is considered to be a standard for such an evaluation. for russian, however, there are significantly fewer evaluation resources. only few tasks from mteb contain russian subsets, while until recently, the only embedding benchmark for russian was enkodechka foot_1 , which appeared several years ago and is still actively used. however, it has significantly fewer tasks than mteb and has no tasks to evaluate the retrieval abilities of the model. this paper addresses both problems by presenting a novel russian-focused embedding model, also adapted for the english language, allowing knowledge transfer from this high resource lan-guage, and a new benchmark for text embedding evaluation in russian called rumteb (see figure 1 for its general structure), which contains 23 text embedding tasks in mteb format. among them, 17 tasks are new, and the other 6 are formed on the multilingual mteb datasets. thus, the contributions of our work are the following: ## text embedding model for russian this section is devoted to the text embedding model ru-en-rosberta released within the research. we describe the training data, the base model, and the final training pipeline, motivated by the experiments described in section 5. ## training procedure analysis this section describes experiments we conducted to determine the final training pipeline. we used basic russian, english, and additional synthetic datasets, the training approach described in section 4 unless otherwise specified, and the full rumteb version for evaluation. details on the model configurations in these experiments are given in a.2.2 and further findings are in appendix a.6. ## evaluation we evaluate ru-en-rosberta and 9 publicly available embedding models for russian, including the multilingual ones and the two instruct models, on the rumteb benchmark. see we evaluate all models in the same environments and scenarios by the procedure described in 3.1. we use mteb framework foot_10 for evaluation where we integrated evaluation on the new rumteb tasks 1819 . model name parameters huggingface hub link citation rubert-tiny2 29.4m cointegrated/rubert-tiny2 -sbert large-nlu-ru 427m ai-forever/sbert_large_nlu_ru -sbert large-mt-nlu-ru 427m ai-forever/sbert_large_mt_nlu_ru ru-en-rosberta 404m ai-forever/ru-en-rosberta -me5 small 118m intfloat/multilingual-e5-small wang et al. (2024) me5 base 278m intfloat/multilingual-e5-base wang et al. (2024) me5 large 560m intfloat/multilingual-e5-large wang et al. (2024) bge-m3 567m baai/bge-m3 multi-granularity me5 large-instruct 560m intfloat/multilingual-e5-large-instruct wang et al. (2024) e5 mistral-7b-instruct 7.11b intfloat/e5-mistral-7b-instruct wang et al. (2023a) ## results as for the non-instruct model analysis, it can be seen that bge-m3, ru-en-rosberta, and me5 large perform practically on par. moreover, ruen-rosberta performs better than its non-instruct competitors on all task categories except for 2 (retrieval and reranking), probably due to the absence of the contrastive pre-train. for retrieval and reranking, bge-m3 and me5 large receive much better scores, resulting in ru-en-rosberta being 20 this results are valid for 09.10.2024. please, refer to the leaderboard at https://huggingface.co/spaces/ mteb/leaderboard for the latest results. in the second place. the evaluation results show that ru-en-rosberta is a robust embedding model suitable for various textual tasks. additionally, unlike monolingual russian models, the bilingual nature of ru-en-rosberta allows it to be further trained or fine-tuned using the much more considerable amount of english data available. the evaluation results positively characterize the benchmark as being complex enough for modern embedding models, allowing researchers to evaluate text embedding at a high level. ## conclusion this paper introduces a new russian-focused embedding model, which also supports english, and a new benchmark for text embedding evaluation, comprising 23 datasets divided into 7 task types. among the benchmark datasets, 17 datasets are new and were created within this research. we report the new embedding model architecture design, pre-training corpus, and training procedure details. we describe the datasets comprising the benchmark and propose the methodology for the text embedding evaluation on it inspired by the mteb benchmark. we evaluate the presented encoding model and several baselines, thus verifying the rumteb complexity and performing the comparative analysis of our model results with the results of standard encoders.