H y QE : Ranking Contexts with Hypothetical Query Embeddings (2024)

Abstract: In retrieval-augmented systems, context ranking techniques are commonly employed to reorder the retrieved contexts based on their relevance to a user query. A standard approach is to measure this relevance through the similarity between contexts and queries in the embedding space. However, such similarity often fails to capture the relevance. Alternatively, large language models (LLMs) have been used for ranking contexts. However, they can encounter scalability issues when the number of candidate contexts grows and the context window sizes of the LLMs remain constrained. Additionally, these approaches require fine-tuning LLMs with domain-specific data. In this work, we introduce a scalable ranking framework that combines embedding similarity and LLM capabilities without requiring LLM fine-tuning. Our framework uses a pre-trained LLM to hypothesize the user query based on the retrieved contexts and ranks the context based on the similarity between the hypothesized queries and the user query. Our framework is efficient at inference time and is compatible with many other retrieval and ranking techniques. Experimental results show that our method improves the ranking performance across multiple benchmarks.

## introduction context retrieval plays a crucial role in natural language processing (nlp). standard techniques can efficiently extract relevant information from dedicated databases to address user queries. these techniques have been driving advancements in search engines, virtual assistants, and other retrievalaugmented systems by enabling precise, real-time responses in real time, and reducing the risk of hallucination @xcite . accurately ranking the relevance of the contexts to the user query is a crucial factor in the performance of retrieval-augmented systems @xcite . classical retrieval methods such as tf-idf and bm25 @xcite rely on lexical similarities to rank contexts. recent advancements in embedding models such as bert @xcite have enabled the capture of the semantic similarity between texts through dense vector representations. to improve the zero-shot performance in unseen contexts, contriever @xcite and other successive embedding models are trained via contrastive learning techniques. however, retrieval with these embedding models focuses on similarity, but similarity alone does not always ensure that the context effectively addresses the query. llms have been incorporated to address this issue. for instance, llm-based re-rankers @xcite can determine whether a context addresses a query better than others. however, those re-rankers require fine-tuning, which demands extensive dedicated datasets and significant computational resources. other methods include using llm to expand the query before retrieval. hyde @xcite , for instance, utilizes an llm to generate hypothetical contexts based on the query, subsequently retrieving concrete contexts that are close to these hypothetical contexts in the embedding space. however, the llm must have sufficient background knowledge about the context to be retrieved so that it can generate semantically similar contexts. otherwise, the hypothesis space of the generated contexts would be indefinitely large, and the llm can generate outdated, irrelevant, hallucinated, and even counterfactual contexts @xcite . we provide an example later in fig. 3(b ), where gpt-3.5-turbo generates outdated information that fails to reflect recent developments on a specific topic. in this paper, we propose a novel context ranking framework. our approach uses an llm to generate hypothetical queries based on the existing contexts. it then measures the relevance between the context and a user-given query based on the similarity between the hypothetical queries and the user-given query. while our method does not require the llm to have prior knowledge about the query or the context, the hallucination of the llm is restrained since a context has limited information and can only provide answers to a certain range of queries. furthermore, while hyde has to use an llm to generate hypothetical contexts online for every input query, our approach allows retrieving previously generated hypothetical queries for future input queries. our method also differs from the llm-based re-ranker in two-fold. first, our method does not require fine-tuning an llm. second, our approach uses text embedding for ranking, while an llm-based re-ranker has to call an llm to answer the relevance between every input query and context. however, our method can be used in conjunction with other ranking methods to iteratively refine the ranking of the retrieved contexts. in addition to introducing our approach, we compare our approach with existing approaches from the theoretical lens. we analyze the causality relationship between the queries and contexts within a class of context ranking approaches, identifying their potential issues, such as their susceptibility to spurious causality relationships. we then show that our approach mitigates these issues by following a variational inference approach. our experimental results demonstrate improvements in ranking the retrieved contexts across multiple information retrieval benchmarks while maintaining efficiency and scalability. our major contribution is listed as follows. retrieval-augmented systems have become a focal point in nlp research, enhancing llms by accessing broader knowledge bases beyond llm context windows @xcite . these systems use information retrieval techniques to fetch relevant contexts from dedicated databases based on user queries, improving performance in tasks requiring extensive context @xcite . information retrieval methods, such as tf-idf and bm25, rely on lexical similarities to rank contexts @xcite . recent advancements in embedding models such as bert @xcite allow capturing text semantics through dense vector representations @xcite . contrastive learning techniques have further improved the zero-shot performance of embedding models such as contriever @xcite in unseen contexts by training the models to differentiate between similar and dissimilar contexts @xcite . document expansion and query expansion are classical techniques to improve retrieval quality and have been widely adopted in rag systems @xcite . query expansion, which dates back to @xcite , typically involves rewriting the query based on labels @xcite . when labels are not available, the query can be expanded with generated contexts @xcite . for instance, hyde @xcite uses llms to generate hypothetical contexts based on the input query and uses the embeddings of the query and the hypothetical contexts for retrieval. however, when the llm lacks knowledge about the query, query expansion can be susceptible to hallucinated or counterfactual content @xcite . document expansion @xcite involves appending each context with a generated query and creating indexes for the expanded context in the database. our framework also generates queries based on the contexts but does not expand the contexts. studies on generating high-quality queries to build synthetic datasets @xcite can be helpful for our framework, but that is not the focus of this paper. large language models (llms), from the smallsize open source models such as mistral-7b @xcite to the large-size proprietary mod-els such as @xcite , are pre-trained on trillions of tokens, exhibiting unparalleled emergent and generalization abilities across tasks @xcite . llms can be fine-tuned to rank the relevancy between contexts and queries @xcite @xcite . although effective, fine-tuning requires significant computational resources and extensive annotated data @xcite . furthermore, those methods have to face the challenges related to the context window size @xcite @xcite , as they combine the query and contexts into a single prompt. our method does not use llms to evaluate the querycontext relevancy. variational inference @xcite sits at the core of our proposed framework. it has been extensively studied across many fields of machine learning @xcite @xcite . in this work, we treat queries and contexts as random variables with causal relationships and reformulate the ranking problem as a probability inference problem. it is widely known that generative models that respect the causality relationships are more robust to distribution shifts because they can avoid learning spurious relationships between random variables @xcite @xcite . in this work, we use an llm to simulate the query-context relationship while avoiding the intervention of prior knowledge, thereby preserving the causal structure. a ranker can just target the first k contexts if the contexts have already been ordered in some previous ranking procedure. we denote the set that includes the first k contexts as c q,k ⊆ c such that |c q,k | = k. after ranking those contexts, a new scoring function r q over c q,k is generated. when using an embedding model for ranking, we use the cosine similarity between query embedding and context embedding to determine the relevance of the query and context. we use e to denote the embedding model. the cosine similarity between a query q and a context c is as a result, given any query q, an embedding model-based ranker's scoring function is defined as r q (c) = sim(q, c). ## method in this section, we introduce our framework for ranking contexts with hypothetical queries. we first illustrate our context ranking procedure, explain how to obtain those hypothetical queries, and then discuss its complexity. for each context @xmath0, we hypothesize the probable queries that the context c can address or the topics it discusses. we refer to these queries as hypothetical queries, denoted as q. for each @xmath1, we let h(c) denote the set of hypothetical queries associated with c. our ranking method determines the relevance of a given query q and a context c based on the similarity between the embedding of q and the embedding of c, as well as the similarity between those of q and the hypothetical queries h(c) as in eq.2 where we introduce a hyperparameter λ to balance the two similarities. algorithm 1 outlines our context ranking procedure. we start with a set c q,k of k candidate contexts, which are typically the top-k results from a prior ranking step. for each context @xmath2,k , we generate a set of hypothetical queries h(c) by using an llm, compute the embedding of c and each @xmath3, and then calculate the relevance score r q (c) using eq.1. hypothetical query generation. our framework allows utilizing various llms ranging from mistral 7b to gpt-3.5 and gpt-4 to generate hypothetical algorithm 1 hyqe 1: input: a query q; a set c q,k of k candidate contexts; an llm h; an embedding model e 2: foreach context c in c q,k 3: compute sim(q, c) via eq.1 4: collect hypothetical queries h(c) 5: compute r q (c) via eq.2 6: order c q,k by r q (c) 7: output: the ordered-set c q,k which kinds of questions can be answered based on the following passage ```< @xmath4, different , and be written on separate lines . if the passage provides no meaningful content , respond with a ' no content '. queries. fig.1(a) shows a flowchart of this query generation process. for each context c, we generate a set of hypothetical queries h(c) by instructing an llm h. specifically, we use a single prompt to generate multiple queries for each context to avoid generating repetitive queries, as shown in fig. 2 . the prompt is designed to ensure that the generated queries are diverse and relevant to the given context. if the length of the context c and the lengths of queries to be generated will exceed the window size of the llm, we partition c, call the llm to generate queries for one portion at a time, and collect all generated queries in the end. complexity. although generating hypothetical queries for each c with an llm can be timeconsuming, this overhead can be mitigated. since the hypothetical queries h(c) are independent of the input q, once h(c) and the corresponding embeddings are obtained, they can be stored and reused for future queries that involve the same context c. this eliminates the repetitive querygeneration step in line 4 of the algorithm. when a previously seen context c is retrieved for some new input query q ′ , we can quickly retrieve the stored h(c) and embeddings of the queries in h(c). then we only need to perform a similarity search to find the hypothetical query @xmath5. by leveraging stored hypothetical queries and their embeddings, our framework ensures efficient and scalable query processing, reducing the computational overhead of real-time llm calls. the complexity of our ranking framework can be broken down as follows. generating hypothetical queries h(c) for each context @xmath6. if each context c can generate m hypothetical queries, the total complexity of this one-time computation is o(|c| • m ) where |c| is the total number of contexts and m is limited by the information encompassed in the context. this complexity is amortized as the number of input queries increases, making our approach more scalable. if a c is retrieved for a new query q ′ and its hypothetical query set h(c) has been indexed, the one-time computational complexity of retrieving the closest hypothetical query @xmath7. in comparison, query expansion methods typically generate contexts for each input query. thus, the total complexity cannot be amortized by the growing number of input queries. for instance, hyde @xcite ) requires generating a group of hypothetical documents for each query, leading to a total complexity of o(|q| • n ) where |q| is the number of queries and n is the number of hypothetical contexts, both of which are independent of the document corpus c and can be infinite. similar comparisons apply to llm-based rerankers @xcite , where the complexity is proportional to the lengths of the input query and the retrieved contexts, as the re-rankers require concatenating the query and contexts in the prompts to generate responses. this complexity cannot be amortized, making ranking contexts expensive as the number of queries increases, considering that llm-based re-rankers are often large, proprietary models. hyqe allows using small pre-trained llms and open-source embedding models, significantly reducing operational costs while maintaining efficiency and effectiveness. ## a variational inference perspective in this section, we explain how to use variational inference to derive eq.2 by establishing the causal relationship between queries and contexts. ## conclusion in this paper, we introduce a novel framework for context ranking using hypothetical queries generated by llms. our method is grounded in variational inference, aiming to preserve the causal relationship between queries and the contexts. the experimental results demonstrate that our approach not only outperforms baselines but also can be integrated seamlessly with existing techniques, allowing for iterative refinement and continuous improvement. furthermore, our method can amortize the overhead in text generation with llm as the input queries increase, offering a scalable and efficient solution for context retrieval and ranking.