D 2 LLM : Decomposed and Distilled Large Language Models for Semantic Search (2024)

Abstract: The key challenge in semantic search is to create models that are both accurate and efficient in pinpointing relevant sentences for queries. While BERT-style bi-encoders excel in efficiency with pre-computed embeddings, they often miss subtle nuances in search tasks. Conversely, GPT-style LLMs with cross-encoder designs capture these nuances but are computationally intensive, hindering real-time applications. In this paper, we present D2LLMs—Decomposed and Distilled LLMs for semantic search—that combines the best of both worlds. We decompose a cross-encoder into an efficient bi-encoder integrated with Pooling by Multihead Attention and an Interaction Emulation Module, achieving nuanced understanding and pre-computability. Knowledge from the LLM is distilled into this model using contrastive, rank, and feature imitation techniques. Our experiments show that D2LLM surpasses five leading baselines in terms of all metrics across three tasks, particularly improving NLI task performance by at least 6.45%

## introduction semantic search has become an integral part of natural language processing, tasked with sifting through extensive texts to find passages that best match a user's query based on underlying semantic links. it transcends the non-semantic techniques, such as tf-idf and bm25, by resolving lexical mismatches and enabling more precise text matching. as a result, semantic search has significant impacts across various fields, including information retrieval @xcite , question answering @xcite , dialogue systems @xcite , item recommendation (hu et al., * equal contribution. this work was done when zihan liao was a research intern at ant group. the major challenge of semantic search lies in devising a model that is both accurate and efficient in pinpointing the most relevant passages for any given query. the current go-to models, particularly the compact bert-style bi-encoders or dual encoders @xcite @xcite , independently convert queries and passages into vectors and judge their relevance through measures such as cosine similarity. this process is praised for efficiency-enabling precomputation and on-the-fly querying of passage vectors. however, this streamlined method comes at an accuracy cost. within the rigidity of the bi-encoder's similarity space, subtle but critical nuances may be lost, such as when differentiating between symmetric search tasks (e.g., finding similar questions to "what are the symptoms of the flu?") and asymmetric search tasks (e.g., matching that same query to a comprehensive answer detailing symptoms). the bi-encoders' constrained interaction mode limits their comprehension of the distinct informational roles that queries and passages play. additionally, bi-encoders are bound to a laborious and multi-stage training process, starting with pretraining on massive datasets of weakly-supervised text pairs and ending with finetuning on diverse and extensive labeled datasets @xcite . this process is heavily data-intensive and usually limited by the variety of data available. moreover, the small size of bi-encoders often means they excel within their training domain but fall short when generalizing to new, unseen domains @xcite . on the flip side, gpt-style large language models (llms) with cross-encoder designs overcome these limitations by jointly processing queries and passages, thereby forming a single, interactive input. this method enables a granular understanding of textual relationships, as it involves concatenat-ing the query and passage, with directive prompts such as "are these questions similar?" or "does this passage answer the question?" to guide the model through both symmetric and asymmetric search tasks. updated llms arrive pre-loaded with a broad spectrum of world knowledge @xcite , eliminating the need for domain-specific pretraining and facilitating rapid adaptation. the remarkable zero-shot learning ability @xcite ensures their robust performance even for novel domains. however, this accurate analysis incurs a toll on computational efficiency; it precludes the caching of passage vectors and necessitates fresh inference for each new querypassage pairing, which can hinder the practicality of cross-encoder llms in situations demanding real-time, voluminous processing. in this paper, we seek to bring the best of both worlds together with the introduction of d2llms, which stands for decomposed and distilled llms for semantic search. our proposed framework begins by decomposing an llm-based cross-encoder into a bi-encoder coupled with an interaction emulation module (iem). the bi-encoder, equipped with pooling by multihead attention (pma) of token embeddings resulting from a pretrained llm, efficiently generates separate embeddings for queries and passages, allowing passage vectors to be pre-stored while ensuring the model's adaptability. the iem goes further, intricately mapping the relationships between queries and passages. it features dedicated branches for handling symmetric and asymmetric search tasks. we then distill the high-level knowledge from the original llm-based cross-encoder (the teacher) into our decomposed model (the student) through a series of teacher-guided methodologies, including contrastive imitation, rank imitation, and feature imitation. our contributions can be summarized as: our d2llm framework overcomes these challenges by focusing on refining the student model through distillation with a static teacher model, which can be pre-finetuned for specific tasks. we enhance the distillation process by employing a combination of contrastive, rank, and feature imitation losses. crucially, we integrate an interaction emulation module into the student model to better understand and replicate the nuanced interplay between queries and passages, thus solving the problems of previous distillation approaches. llms classical semantic search methods often suffer from limited size, leading them to excel in one specific task like text similarity, classification, or information retrieval, as with simcse @xcite and contriever @xcite . yet, larger models and improved pretraining techniques have been shown to significantly boost both the accuracy and applicability of plm-based dense retrieval @xcite @xcite . this has led to the application of llms to semantic search, though this field remains underexplored. current approaches to transforming llms into bi-encoders include finetuning only methods like sgpt @xcite , re-pllama @xcite , udever @xcite , and llara @xcite , as well as those combining continued pretraining with finetuning, such as cpt @xcite and gtr @xcite . typically, these methods introduce a special token at the end of the passage, trained with contrastive loss to emulate the role of bert's [cls] token. however, this adaptation deviates from the original training objective of llms, which is next-token prediction, potentially underutilizing llms' capabilities and resulting in suboptimal performance compared to smaller, specialized bidirectional encoders like gte @xcite and bge @xcite . the proposed d2llm tackles this issue by decomposing the teacher llm into a student biencoder combined with an interaction emulation module that more faithfully mirrors the llm's handling of query-passage interactions. through a detailed distillation process, d2llm effectively transfers knowledge from the teacher llm, producing a student model that excels beyond the aforementioned models finetuned on the same datasets. ## d2llm the architecture of d2llm is depicted in figure 1 . here, the teacher model operates as an llmbased cross-encoder, adeptly handling joint querypassage inputs to leverage the llm's robust semantic comprehension for more accurate outcomes. on the other hand, the student model features a bi-encoder for pre-computing passage vectors to ensure operational efficiency, complemented by an interaction emulation module that considers the complex query-passage interplay. through an elaborate knowledge distillation procedure, we aim to cultivate a student model that emulates the teacher's accuracy while retaining efficiency. subsequently, we detail the process of decomposing the teacher model to assemble the student model and elucidate the knowledge distillation methodology. ## experiments in this section, we evaluate d2llm's effectiveness on three tasks: natural language inference (nli), semantic textual similarity (sts), and information retrieval (ir), with a focus on chinese. the former two are symmetric search tasks, and the last is asymmetric. details on datasets and evaluation metrics are provided in appendix a. for nli and sts, we use 0.3 million training samples, and 0.9 million for ir. the proposed method (denoted as d2llm 1 ) is compared with five stateof-the-art baselines: bge @xcite , rocketqav2 @xcite , sgpt (muennighoff, 2022), udever @xcite , and llara @xcite . bge and rocketqav2 are bert-style bi-encoders; bge uses pretraining and finetuning, while rocketqav2 distills from a cross-encoder, updating both the student and teacher. the remaining three finetune gpt-style llms into bi-encoders. for more details, please refer to appendix c. the total number of parameters for each method is shown in parentheses after it. to ensure a fair comparison, all baseline methods have a number of tunable parameters (denoted as #param. in tables 1 2 ) equal to, or greater than, those in d2llm, with the exception of sgpt, which regards the use of bitfit finetuning @xcite as its notable advantage. all methods, except sgpt, incorporate the hard negatives from we also use chinese-roberta-large as the base for rocketqav2. for the other methods, we select qwen-7b-chat @xcite as the base model. additionally, the performance of the cross-encoder teacher model based on qwen-7b-chat (i.e., llmce) is presented, as well as a bi-encoder (i.e., llmbe) that generates sentence embeddings by meanpooling token embeddings from the last layer of qwen-7b-chat, in order to gauge d2llm's improvement over the untrained llm-be and its proximity to the teacher llm-ce's performance. ## conclusion this paper presents d2llm, an innovative model distilling knowledge from an llm teacher to construct an efficient student for semantic search. with a deep and nuanced understanding of its teacher, d2llm utilizes specially designed modules and losses to encapsulate the teacher's prowess in a more compact form. our experimental findings reveal that d2llm successfully synthesizes the high accuracy associated with cross-encoders and the operational efficiency of bi-encoders.