Matryoshka-Adaptor: Unsupervised and Supervised Tuning for Smaller Embedding Dimensions (2024)

Abstract: Embeddings from Large Language Models (LLMs) have emerged as critical components in various applications, particularly for information retrieval. While high-dimensional embeddings generally demonstrate superior performance as they contain more salient information, their practical application is frequently hindered by elevated computational latency and the associated higher cost. To address these challenges, we propose Matryoshka-Adaptor, a novel tuning framework designed for the customization of LLM embeddings. Matryoshka-Adaptor facilitates substantial dimensionality reduction while maintaining comparable performance levels, thereby achieving a significant enhancement in computational efficiency and cost-effectiveness. Our framework directly modifies the embeddings from pre-trained LLMs which is designed to be seamlessly integrated with any LLM architecture, encompassing those accessible exclusively through black-box APIs. Also, it exhibits efficacy in both unsupervised and supervised learning settings. A rigorous evaluation conducted across a diverse corpus of English, multilingual, and multimodal datasets consistently reveals substantial gains with Matryoshka-Adaptor. Notably, with Google and OpenAI Embedding APIs, Matryoshka-Adaptor achieves a reduction in dimensionality ranging from two- to twelve-fold without compromising performance across multiple BEIR datasets.

## introduction large language models (llms) have showcased remarkable proficiency in handling various text processing tasks, encompassing question answering, summarization, and mathematical reasoning @xcite @xcite . this success can be partially attributed to their ability to transform raw text into semantically enriched representations, with the qual-12x dimension reduction 53.9% performance gains 2x dimension reduction 16.2% performance gains figure 1 : the effectiveness of the matryoshka adaptor in dimensionality reduction. in both unsupervised (red line) and supervised (black line) settings, the matryoshka adaptor showcases the capability to considerably decrease embedding dimensions while maintaining a negligible impact on ndcg@10 retrieval performance with beir scifact dataset. notably, at the same embedding dimensionality, the utilization of our approach results in significantly improved performance. embeddings find extensive utilization in a wide array of downstream tasks, with information retrieval (ir) serving as a prominent one @xcite @xcite . ir involves the process of searching for relevant information within a corpus database using queries, and language modeling plays a pivotal role as both queries and corpus data are frequently textual in nature. in ir systems, text embeddings are commonly employed to rank relevant corpora based on their similarity to queries. numerous llms have been developed specifically for extracting embeddings from raw text, including sentence t5 @xcite , openai embedding apis (ope), and google embedding apis (gcp). however, generation of high-dimensional embeddings often entail high latency and computational costs, thereby limiting their practical application in latency-sensitive scenarios such as largescale recommendation systems. matryoshka representation learning (mrl) @xcite addresses this limitation at pretraining stage, by generating embeddings that retain similar characteristics even when utilizing only a subset of their dimensions (i.e., matryoshka properties). this enables efficient similarity comparisons with lower-dimensional embeddings and is leveraged by many state-of-the-art models, including those from google and openai @xcite . in this study, we present matryoshka-adaptor, a novel framework designed to transform arbitrary embeddings into embeddings with matryoshka properties in both unsupervised and supervised learning setups. in an unsupervised learning setting, the adaptor learns to transform input embeddings into matryoshka embeddings using only corpus embeddings. we introduced pairwise and top-k similarity loss functions to facilitate this process. in supervised learning setup, the adaptor can be further refined by incorporating relevant (query, corpus) pairs as labeled tuning data. note that we customize embeddings to better suit specific datasets, using both unsupervised and supervised approaches. this refinement results in improved matryoshka properties, surpassing even those of embeddings derived from mrl-trained models. extensive experiments conducted across 13 beir datasets @xcite , 17 mir-acl datasets @xcite , and 5 multimodal fashion-200k datasets @xcite validate the effectiveness of matryoshka-adaptor. we demonstrate significant performance improvements over the latest google (for english only, multilingual, and multimodal data) @xcite and openai text embedding models (ope). matryoshka-adaptor exhibits broad applicability to diverse embedding types, encompassing text, multimodal, multilingual, and use case-specific embeddings. the contributions of this paper are threefold: ## conclusions while high-dimensional embeddings often provide a richer representation of the original data, their practical deployment in real-world applications is frequently hampered by computational costs and latency issues. as a result, many applications opt for lower-dimensional embeddings, accepting a compromise in performance. the presented matryoshka-adaptor framework offers a solution to this dilemma by enabling substantial dimensionality reduction while minimizing performance degradation, in both supervised and unsupervised scenarios. furthermore, due to its versatile nature and applicability to any embedding model, regardless of access to model parameters, the matryoshka-adaptor constitutes a valuable tool for improving the efficiency and practicality of embedding-based applications.