TR - MTEB : A Comprehensive Benchmark and Embedding Model Suite for T urkish Sentence Representations (2025)

Abstract: We introduce TR-MTEB, the first large-scale, task-diverse benchmark designed to evaluate sentence embedding models for Turkish. Covering six core tasks as classification, clustering, pair classification, retrieval, bitext mining, and semantic textual similarity, TR-MTEB incorporates 26 high-quality datasets, including native and translated resources. To complement this benchmark, we construct a corpus of 34.2 million weakly supervised Turkish sentence pairs and train two Turkish-specific embedding models using contrastive pretraining and supervised fine-tuning. Evaluation results show that our models, despite being trained on limited resources, achieve competitive performance across most tasks and significantly improve upon baseline monolingual models. All datasets, models, and evaluation pipelines are publicly released to facilitate further research in Turkish natural language processing and low-resource benchmarking.

## introduction text embeddings form the backbone of modern natural language processing (nlp) systems, enabling efficient and scalable representations of text for a wide range of applications, from retrieval and classification to clustering and summarization. recent advances have demonstrated that universal embedding models can power diverse downstream tasks with minimal adaptation, making them a critical component in the era of large-scale language technologies. however, the evaluation and development of such models have largely centered around high-resource languages like english, while lowresource, morphologically complex languages such as turkish remain underrepresented. benchmarks like the massive text embedding benchmark (mteb) @xcite 1 tr-mteb github repository have played a pivotal role in standardizing the evaluation of text embeddings across multiple tasks and domains. following the success of mteb, language-specific extensions such as c-mteb @xcite for chinese and pl-mteb @xcite for polish have shown that adapting benchmarks to linguistic and cultural contexts leads to deeper insights into model performance. yet, no such comprehensive resource currently exists for turkish, limiting both the development of robust turkish embedding models and the fair evaluation of multilingual systems on turkish text. in this work, we introduce the turkish massive text embedding benchmark (tr-mteb), the first large-scale, standardized benchmark specifically designed to evaluate sentence embeddings for turkish across six key tasks: classification, clustering, retrieval, bitext mining, pair classification, and semantic textual similarity. tr-mteb draws from a diverse mixture of native turkish datasets and highquality translated resources, including carefully adapted retrieval datasets from the beir @xcite benchmark suite. all translations underwent rigorous quality control to ensure semantic fidelity and task alignment. beyond benchmarking, we address the critical shortage of high-quality turkish training data for embedding models. inspired by methodologies from the bge @xcite , gte @xcite and e5 @xcite ) models, we curated a corpus of 34.2 million weakly supervised turkish sentence pairs sourced from publicly available data. using this corpus, we pretrained general text embedding models for turkish from berturk @xcite base model, and additionally trained these models using labeled data to see the effects of fine-tuning to assess their performance on tr-mteb. our contributions are: (1) tr-mteb, a taskdiverse turkish benchmark for evaluating sentence embeddings across six core nlp tasks; (2) a highquality corpus of 34.2 million weakly supervised turkish sentence pairs curated from over 50 public datasets; (3) newly trained turkish-specific embedding models using contrastive pretraining and supervised fine-tuning on this corpus; and (4) publicly released datasets, models, and evaluation pipelines to support future turkish nlp research and encourage equitable benchmarking for other low-resource languages. ## related work recent years have seen the emergence of largescale benchmarks like the massive text embedding benchmark (mteb), which provides a unified framework for evaluating sentence embeddings across a wide range of nlp tasks, including classification, clustering, retrieval, and semantic similarity. building on this foundation, languagespecific adaptations such as c-mteb for chinese, pl-mteb for polish, @xcite , rumteb @xcite for russian, and scandinavian embedding benchmark @xcite have demonstrated the importance of tailoring benchmarks to linguistic nuances and resource availability. however, for the turkish language, efforts have been relatively limited. mukayese @xcite has made progress by introducing taskspecific benchmarks for turkish nlp, including classification, translation, and summarization, but it does not evaluate sentence embeddings in a broad range of tasks. additionally, earlier intrinsic evaluations of turkish word embeddings @xcite highlighted the challenges posed by turkish morphology, but they remained at the word level and lacked task diversity. the beir benchmark @xcite , widely used for zero-shot retrieval evaluation, has also inspired multilingual adaptations like beir-pl @xcite and beir-nl @xcite , which rely on translated datasets to support information retrieval in polish and dutch, respectively. these works show that translation, when combined with native resources, can effectively bootstrap evaluation frameworks for lowresource languages. in addition to the development of benchmarks, recent advancements in embedding models have emphasized the importance of large-scale, weakly supervised training data. the c-pack framework @xcite exemplifies this approach by introducing c-mtp, a massive dataset comprising 100 million chinese text pairs extracted from diverse sources such as web corpora, questionanswer (qa) forums, and encyclopedic content. these pairs were utilized to train the baai general embeddings (bge) models, which demonstrated significant performance improvements across various tasks in the c-mteb benchmark. our work builds on these foundations by introducing tr-mteb, the first task-diverse benchmark for evaluating sentence embeddings in turkish. like c-mteb and pl-mteb, we combine native corpora with carefully translated datasets, but we also go further by training and evaluating turkish-specific embedding models, thus addressing a critical gap in turkish nlp research. similar to c-mtp, our study involves the creation of 34.2 million weakly supervised turkish sentence pairs from various sources. these pairs were instrumental in training turkish-specific embedding models, with the aim of improving performance across the tasks defined in the tr-mteb benchmark. this approach aligns with the strategies employed in the c-pack and e5 frameworks, underscoring the efficacy of large-scale, diverse training data in developing versatile embedding models. in parallel to these efforts, the massive multilingual text embedding benchmark (mmteb) has recently been introduced as a large-scale, community-driven expansion of mteb @xcite . covering more than 500 tasks across over 250 languages, mmteb broadens the scope of embedding evaluation by incorporating novel and challenging tasks such as instruction following, long-document retrieval, and code retrieval. to mitigate the computational cost of such a massive benchmark, the authors propose correlationbased downsampling and optimized retrieval splits with hard negative sampling, while still preserving model ranking fidelity. this concurrent work highlights the growing importance of multilingual and large-scale benchmarking for embeddings. in contrast, our tr-mteb benchmark focuses specifically on turkish, addressing linguistic challenges such as rich morphology and limited resources, thereby complementing the multilingual perspective of mmteb with a fine-grained, languagespecific evaluation framework. ## methodology in this section, we present our methodology for developing and evaluating turkish sentence embedding models. our approach consists of three main components: (1) the construction of a comprehensive turkish benchmark for embedding evaluation, (2) the collection and curation of a largescale weakly supervised text pair corpus for model training, and (3) the training of turkish-specific embedding models using modern contrastive learning techniques. together, these components aim to establish a robust evaluation and training framework tailored to the linguistic characteristics and resource limitations of turkish. ## experiments & results to evaluate sentence embedding models for turkish, we benchmarked a diverse set of multilingual and monolingual encoders on the tr-mteb benchmark. ## conclusion this paper introduced tr-mteb, the first largescale benchmark for evaluating turkish sentence embeddings across six task types and 26 datasets, combining native and high-quality translated resources. to support training, we compiled a 34.2m weakly supervised sentence pairs corpus and used it to train two turkish-specific embedding models based on bert-base-turkish-uncased: one trained with cached-mnrl for efficient contrastive learning, and one further fine-tuned on tr-mteb training splits. our models substantially outperform the base berturk model and perform competitively on monolingual tasks such as classification and sts, despite being trained with significantly less data and compute than large multilingual counterparts. the remaining gap in cross-lingual tasks, notably bitext mining, highlights the limitations of monolingual-only training for multilingual alignment. as future work, we plan to focus on scaling the training data, incorporating multilingual contrastive objectives to improve cross-lingual generalization, and extending tr-mteb with additional tasks such as reranking and summarization to enable more comprehensive evaluation.