M 3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation (2024)

Abstract: In this paper, we introduce a new embedding model called M3-Embedding, which is distinguished for its versatility in Multi-Linguality, Multi-Functionality, and Multi-Granularity. It provides a uniform support for the semantic retrieval of more than 100 working languages. It can simultaneously accomplish the three common retrieval functionalities: dense retrieval, multi-vector retrieval, and sparse retrieval. Besides, it is also capable of processing inputs of different granularities, spanning from short sentences to long documents of up to 8,192 tokens. The effective training of M3-Embedding presents a series of technical contributions. Notably, we propose a novel self-knowledge distillation approach, where the relevance scores from different retrieval functionalities can be integrated as the teacher signal to enhance the training quality. We also optimize the batching strategy, which enables a large batch size and high training throughput to improve the discriminativeness of embeddings. M3-Embedding exhibits a superior performance in our experiment, leading to new state-of-the-art results on multilingual, cross-lingual, and long-document retrieval benchmarks.

## introduction embedding models are a critical form of dnn application in natural language processing. they encode the textual data in the latent space, where the underlying semantics of the data can be expressed by the output embeddings @xcite . with the advent of pre-trained language models, the quality of text embeddings have been substantially improved, making them imperative components for the information retrieval (ir) system. one common form of embedding-based ir application is dense retrieval, where relevant answers to the query can be retrieved based on the embedding similarity @xcite @xcite @xcite . besides, the embedding model can also be applied to other ir tasks, such as multi-vector retrieval where the fine-grained relevance between query and document is computed based on the interaction score of multiple embeddings @xcite , and sparse or lexical retrieval where the importance of each term is estimated by its output embedding @xcite @xcite . despite the widespread popularity of text embeddings, the existing methods are still limited in versatility. first of all, most of the embedding models are tailored only for english, leaving few viable options for the other languages. secondly, the existing embedding models are usually trained for one single retrieval functionality. however, typical ir systems call for the compound workflow of multiple retrieval methods. thirdly, it is challenging to train a competitive long-document retriever due to the overwhelming training cost, where most of the embedding models can only support short inputs. to address the above challenges, we introduce m3-embedding, which is pronounced for its breakthrough of versatility in working languages, retrieval functionalities, and input granularities. particularly, m3-embedding is proficient in multi-linguality, which is able to support more than 100 world languages. by learning a common semantic space for different languages, enables both multilingual retrieval within each language and crosslingual retrieval between different languages. besides, it is able to generate versatile embeddings to support different retrieval functionalities, not just dense retrieval, but also sparse retrieval and multivector retrieval. finally, m3-embedding is learned to process different input granularities, spanning from short inputs like sentences and passages, to long documents of up to 8,192 input tokens. the training of m3-embedding poses a significant challenge. in our work, the following technical contributions are made to optimize the embedding quality. firstly, we propose a novel self knowledge distillation framework, where the multiple retrieval functionalities can be jointly learned and mutually reinforced. in m3-embedding, the @xcite embedding is used for dense retrieval, while embeddings from other tokens are used for sparse retrieval and multi-vector retrieval. based on the principle of ensemble learning @xcite , such heterogenous predictors can be combined as a stronger predictor. thus, we integrate the relevance scores from different retrieval functions as the teacher signal, which is used to enhance the learning process via knowledge distillation. secondly, we optimize the batching strategy to achieve a large batch size and high training throughput, which substantially contributes to the discriminativeness of embeddings. last but not least, we perform extensive and high-quality data curation. our dataset includes three sources: 1) the extraction of unsupervised data from massive multi-lingual corpora, 2) the integration of closely related supervised data, 3) the synthesization of scarce training data. the three data sources are complement to each other and applied to different training stages, which lays a solid foundation for the versatile text embeddings. m3-embedding exhibits a remarkable versatility in our experiments. it achieves superior retrieval quality for a variety of languages, leading to state-of-the-art performances on popular multilingual and cross-lingual benchmarks like mir-acl @xcite and mkqa @xcite . it effectively learns the three retrieval functionalities, which can not only work individually but also work together for an even stronger retrieval quality. it also well maintains its superior capability across different input granularities within 8192 tokens, which outperforms the existing methods by a notable advantage. our contributions are summarized as follows. 1) we present m3-embedding, which achieves unprecedented versatility in multi-linguality, multifunctionality, and multi-granularity. 2) we propose a novel training framework of self-knowledge distillation and optimize the batching strategy for efficient training. we also create high-quality training resource based on comprehensive data curation. 3) our model, code, and data is publicly available, offering critical resources for both direct usage and future development of text embeddings. ## related work the related works are reviewed from three aspects: general text embeddings, embedding models for neural retrieval, embeddings of multi-linguality. in the past few years, substantial progress has been achieved in the field of text embedding. one major driving force is the popularity of pre-trained language models, where the underlying semantic of the data can be effectively encoded by such powerful text encoders @xcite @xcite . in addition, the progress of contrastive learning is another critical factor, especially the improvement of negative sampling @xcite and the exploitation of knowledge distillation @xcite @xcite . on top of these well-established techniques, it becomes increasingly popular to learn versatile embedding models, which are able to uniformly support a variety of application scenarios. so far, there have been many impactful methods in the direction, like contriever @xcite , llm-embedder @xcite , e5 @xcite , bge @xcite , sgpt (muennighoff, 2022), and open text embedding @xcite , which significantly advance the usage of text embeddings for general tasks. one major application of embedding models is neural retrieval @xcite . by measuring the semantic relationship with the text embeddings, the relevant answers to the input query can be retrieved based on the embedding similarity. the most common form of embedding-based retrieval method is dense retrieval @xcite , where the text encoder's outputs are aggregated (e.g., via @xcite or mean-pooling) to compute the embedding similarity. another common alternative is known as multi-vecor retrieval @xcite , which applies fine-grained interactions for the text encoder's outputs to compute the embedding similarity. finally, the text embeddings can also be transformed into term weights, which facilitates sparse or lexical retrieval @xcite @xcite . typically, the above retrieval methods are realized by different embedding models. to the best of our knowledge, no existing method is able to unify all these functionalities. despite the substantial technical advancement, most of the existing text embeddings are developed only for english, where other languages are lagging behind. to mitigate this problem, continual efforts are presented from multiple directions. one is the development of pre-trained multi-lingual text encoders, such as mbert @xcite , mt5 @xcite , xlm-r @xcite . another one is the curation of training and evaluation data for multi-lingual text embeddings, e.g., miracl @xcite , mmarco @xcite , mr. tydi @xcite , mkqa @xcite . at the same time, the multi-lingual text embeddings are continually developed from the community, e.g., mdpr @xcite , mcontriever @xcite , me5 @xcite , etc. however, the current progress is still far from enough given the notable gap with english models and the huge imbalance between different languages. ## m3-embedding m3-embedding realizes three-fold versatility. it supports a wide variety of languages and handles input data of different granularities. besides, it unifies the common retrieval functionalities of text embeddings. formally, given a query q in an arbitrary language x, it is able to retrieve document d in language y from the corpus d y : d y ← fn * (q x , d y ). in this place, fn * (•) belongs to any of the functions: dense, lexical, or multi-vector retrieval; y can be another language or the same language as x. ## experiment in this section, we investigate m3-embedding's performance in terms of multi-lingual retrieval, crosslingual retrieval, and long-doc retrieval. we also explore the impact of its technical factors. ## conclusion in this paper, we introduce m3-embedding, which substantially advances the versatility of text embeddings in terms of supporting multi-lingual retrieval, handling input of diverse granularities, and unifying different retrieval functionalities. m3-embedding presents three technical contributions: self-knowledge distillation, efficient batching, and high-quality curation of data. the effectiveness of m3-embedding is empirically verified, where it leads to superior performances on multi-lingual retrieval, cross-lingual retrieval, and multi-lingual long-document retrieval tasks.