MTEB : Massive Text Embedding Benchmark (2023)

Abstract: Text embeddings are commonly evaluated on a small set of datasets from a single task not covering their possible applications to other tasks. It is unclear whether state-of-the-art embeddings on semantic textual similarity (STS) can be equally well applied to other tasks like clustering or reranking. This makes progress in the field difficult to track, as various models are constantly being proposed without proper evaluation. To solve this problem, we introduce the Massive Text Embedding Benchmark (MTEB). MTEB spans 8 embedding tasks covering a total of 58 datasets and 112 languages. Through the benchmarking of 33 models on MTEB, we establish the most comprehensive benchmark of text embeddings todate. We find that no particular text embedding method dominates across all tasks. This suggests that the field has yet to converge on a universal text embedding method and scale it up sufficiently to provide state-of-theart results on all embedding tasks. MTEB comes with open-source code and a public leaderboard at https://github.com/embeddings-benchmark/mteb .

## introduction natural language embeddings power a variety of use cases from clustering and topic representation @xcite to search systems and text mining @xcite @xcite to feature representations for downstream models @xcite . using generative language models or cross-encoders for these applications is often intractable, as they may require exponentially more computations @xcite . however, the evaluation regime of current text embedding models rarely covers the breadth of their possible use cases. for example, sim-cse @xcite or sbert (reimers and * most of the work done while at hugging face gurevych, 2019) solely evaluate on sts and classification tasks, leaving open questions about the transferability of the embedding models to search or clustering tasks. sts is known to poorly correlate with other real-world use cases @xcite . further, evaluating embedding methods on many tasks requires implementing multiple evaluation pipelines. implementation details like pre-processing or hyperparameters may influence the results making it unclear whether performance improvements simply come from a favorable evaluation pipeline. this leads to the "blind" application of these models to new use cases in industry or requires incremental work to reevaluate them on different tasks. the massive text embedding benchmark (mteb) aims to provide clarity on how models perform on a variety of embedding tasks and thus serves as the gateway to finding universal text embeddings applicable to a variety of tasks. mteb consists of 58 datasets covering 112 languages from 8 embedding tasks: bitext mining, classification, clustering, pair classification, reranking, retrieval, sts and summarization. mteb software is available open-source foot_0 enabling evaluation of any embedding model by adding less than 10 lines of code. datasets and the mteb leaderboard are available on the hugging face hub foot_1 . we evaluate over 30 models on mteb with additional speed and memory benchmarking to provide a holistic view of the state of text embedding models. we cover both models available open-source as well as models accessible via apis, such as the openai embeddings endpoint. we find there to be no single best solution, with different models dominating different tasks. our benchmarking sheds light on the weaknesses and strengths of individual models, such as simcse's @xcite ) low performance on clustering and retrieval despite its strong performance on sts. we hope our work makes selecting the right embedding model easier and simplifies future embedding research. ## conclusion in this work, we presented the massive text embedding benchmark (mteb). consisting of 8 text embedding tasks with up to 15 datasets each and covering 112 languages, mteb aims to provide reliable embedding performance estimates. by opensourcing mteb alongside a leaderboard, we provide a foundation for further pushing the state-ofthe-art of available text embeddings. to introduce mteb, we have conducted the most comprehensive benchmarking of text embeddings to date. through the course of close to 5,000 experiments on over 30 different models, we have set up solid baselines for future research to build on. we found model performance on different tasks to vary strongly with no model claiming state-ofthe-art on all tasks. our studies on scaling behav-ior, model efficiency and multilinguality revealed various intricacies of models that should ease the decision-making process for future research or industry applications of text embeddings. we welcome task, dataset or metric contributions to the mteb codebase foot_6 as well as additions to the leaderboard via our automatic submission format foot_7 .