F a MTEB : Massive Text Embedding Benchmark in P ersian Language (2025)

Abstract: In this paper, we introduce a comprehensive benchmark for Persian (Farsi) text embeddings, built upon the Massive Text Embedding Benchmark (MTEB). Our benchmark includes 63 datasets spanning seven different tasks: classification, clustering, pair classification, reranking, retrieval, summary retrieval, and semantic textual similarity. The datasets are a combination of existing, translated, and newly generated (synthetic) data, offering a diverse and robust evaluation framework for Persian language models. All newly translated and synthetic datasets were rigorously evaluated by both humans and automated systems to ensure high quality and reliability. Given the growing adoption of text embedding models in chatbots, evaluation datasets are becoming an essential component of chatbot development and Retrieval-Augmented Generation (RAG) systems. As a contribution, we include chatbot evaluation datasets in the MTEB benchmark for the first time. Additionally, we introduce the novel task of summary retrieval, which is not included in the standard MTEB tasks. Another key contribution of this work is the introduction of a substantial number of new Persian-language NLP datasets for both training and evaluation, many of which have no existing counterparts in Persian. We evaluate the performance of several Persian and multilingual embedding models across a wide range of tasks. This work presents an open-source benchmark with datasets, accompanying code, and a public leaderboard.

## introduction text-embedding models aim to generate a semantic vector representation of text, which is helpful in addressing various natural language processing (nlp) tasks such as clustering, classification, semantic textual similarity (sts), information retrieval (ir), and more @xcite . to evaluate the performance of models on these tasks, most existing benchmarks are task-specific and fail to assess model capabilities across multiple tasks. for instance, an information retrieval model like dense passage retrieval (dpr) @xcite ) may perform well on retrieval tasks but fails to achieve satisfactory results in finding semantic textual similarity (sts). to address this limitation, the massive text embedding benchmark (mteb) @xcite introduced a comprehensive evaluation suite across eight diverse nlp tasks, effectively meeting the evaluation needs of text-embedding models for the english language. however, since the primary focus of this benchmark is on english, it does not adequately assess the performance of models in low-resource languages like persian. in this work, we introduce famteb, a largescale persian benchmark for evaluating persian text-embedding models, enabling users to select the most suitable text-embedding model for their specific tasks. this benchmark comprises 63 datasets across 7 tasks, and we compare the performance of 15 existing persian or multilingual language models on it. of these, 24 datasets were pre-existing in persian, while we contribute 39 new datasets. the newly introduced datasets were developed using three distinct approaches: web-based collection (4 datasets), translation of existing english datasets (16 datasets), and generation through large language models (llms) as synthetic datasets (19 datasets). the quality of all datasets has been independently assessed to ensure reliability. to assess the generalizability of text-embedding models across various nlp tasks, it is essential to conduct a comprehensive evaluation on diverse problems. since one of the primary and recently highly popular applications of text-embedding models is in retrieval-augmented generation (rag) systems and chatbots, a portion of the newly curated datasets is dedicated to evaluating these systems. this aspect is explored for the first time within this benchmark. needless to say that the persian language is among the low-resource languages in the web and gathering relevant and high quality data is not readily available. below we highlight the main contributions of this work: ## conclusion in this study, we presented famteb, a comprehensive benchmark for evaluating persian text embeddings, building upon the massive text embedding benchmark (mteb). our contribution spans several dimensions, including the introduction of 63 datasets across seven diverse tasks, the addition of a novel task (summary retrieval), and the creation of numerous new persian-language nlp datasets for training and evaluation. by integrating datasets related to chatbot challenges and rag systems into mteb for the first time, we provide a tailored evaluation framework that aligns with emerging applications of text embedding models. the performance evaluation of persian and multilingual embedding models further underscores the benchmark's utility. this open-source benchmark, encompassing datasets, code and a public leaderboard, establishes a robust foundation for advancing nlp research in the persian language. we hope famteb inspires further innovation and enhances the development of more effective persianlanguage models. as part of future work, we plan to reduce the size of selected beir datasets and explore the use of closed-source llms such as gpt-4o-mini, as well as powerful open-source alternatives like gemma-3-27b-it, for translation purposes. this will improve the quality of dataset translations. additionally, we intend to expand the benchmark by including results from a wider range of embedding models, particularly those based on llms, to enable a more comprehensive evaluation across architectures and training paradigms.