Enhancing Conversational Search: Large Language Model-Aided Informative Query Rewriting (2023)

Abstract: Query rewriting plays a vital role in enhancing conversational search by transforming context-dependent user queries into standalone forms. Existing approaches primarily leverage human-rewritten queries as labels to train query rewriting models. However, human rewrites may lack sufficient information for optimal retrieval performance. To overcome this limitation, we propose utilizing large language models (LLMs) as query rewriters, enabling the generation of informative query rewrites through well-designed instructions. We define four essential properties for well-formed rewrites and incorporate all of them into the instruction. In addition, we introduce the role of rewrite editors for LLMs when initial query rewrites are available, forming a “rewrite-then-edit” process. Furthermore, we propose distilling the rewriting capabilities of LLMs into smaller models to reduce rewriting latency. Our experimental evaluation on the QReCC dataset demonstrates that informative query rewrites can yield substantially improved retrieval performance compared to human rewrites, especially with sparse retrievers.

## introduction conversational search has gained significant prominence in recent years with the proliferation of digital virtual assistants and chatbots, enabling users to engage in multiple rounds of interactions to obtain information @xcite @xcite . this emerging search paradigm offers remarkable advantages in assisting users with intricate information needs and complex tasks @xcite . however, a fundamental challenge in conversational search lies in accurately determining users' current search intents within the conversational context. an effective approach that has gained increasing attention addresses this challenge of conversational 1 our implementation is available at: https://github. com/smartyfh/infocqr . what job did elizabeth blackwell have? in what field? she was a lecturer. she was a lecturer in midwifery. did she do well? did elizabeth blackwell do well? did elizabeth blackwell do well as a lecturer in midwifery? ## task formulation the primary objective of conversational search is to identify relevant passages from a vast collection of passages in response to the current user query. formally, let q i and a i be the user query and system response at turn i, respectively. furthermore, let x @xmath0, a 1 , . . . , q t-1 , a t-1 } represent the conversational context up to turn t. then, the task of conversational search can be formulated as retrieving top-k relevant passages, denoted as r k , from a large passage collection c given the current user query q t and its associated context x t . this retrieval process is accomplished by a retriever defined as f : (q t , x t , c) → r k , where r k is a subset of c and k is considerably smaller than the total number of passages in c. the unique challenge in conversational search is incorporating the conversational context while retrieving relevant passages, which cannot be directly addressed by existing retrievers designed for standalone queries. additionally, re-training retrievers tailored for conversational queries can be expensive or even infeasible due to complex system designs or limited data availability @xcite . to overcome the need for re-training, query rewriting is employed as an effective solution @xcite . query rewriting involves transforming the context-dependent user query q t into a self-contained standalone query q ′ t by extracting relevant information from the context x t . consequently, any existing off-the-shelf retrieval systems designed for standalone queries can be leveraged by taking q ′ t as the input query to find passages that are relevant to the original user query q t , i.e., f : (q ′ t , c) → r k . the utilization of query rewriting shifts the challenge of modeling conversational context from the retriever end to the query rewriting model end. as given a question and its context and a rewrite that decontextualizes the question, edit the rewrite to create a revised version that fully addresses coreferences and omissions in the question without changing the original meaning of the question but providing more information. the new rewrite should not duplicate any previously asked questions in the context. if there is no need to edit the rewrite, return the rewrite as-is. ## approach in contrast to relying on human annotators to generate more informative rewrites or developing more complex models to closely replicate existing human rewrites, we propose to prompt llms to generate informative query rewrites simply by providing clear instructions and appropriate demonstrations, avoiding the requirement for extensive human effort and complicated model designs. figure 2 illustrates our proposed approach. ## related work conversational search addresses users' information needs through iterative interactions @xcite . it allows users to provide and seek clarifications @xcite and explore multiple aspects of a topic, thereby excelling at fulfilling complex information needs. the primary challenge in conversational search is accurately identifying users' search intents from their contextualized and potentially ambiguous queries @xcite @xcite @xcite . most existing work @xcite @xcite @xcite addresses this challenge by regarding the concatenation of the current user query with its associated conversational context as a standalone query. however, using this concatenation directly as input to search systems could result in poor retrieval performance @xcite . moreover, this approach requires training specialized retrievers such as dual encoders @xcite @xcite , which can be challenging or even impractical in many real-world scenarios @xcite . another line of research addresses this challenge through query rewriting @xcite @xcite @xcite , which converts the original query into a standalone one. however, these approaches mainly rely on human rewrites to train query rewriting models. as shown in our experiments, human rewrites may lack sufficient informativeness, hence leading to sub-optimal performance of these rewriting models. alternatively, some studies employ query expansion to address this challenge. they select relevant terms from the conversational context @xcite or generate potential answers @xcite to augment the original query. the latter can be seamlessly integrated into our approach to leverage the knowledge within llms. we leave this study as future work. ## conclusion in this work, we propose to prompt llms as query rewriters and rewrite editors for informative query rewrite generation. we are the first to introduce the concept of informative query rewriting and identify four properties that characterize a well-formed rewrite. we also propose distilling the rewriting capabilities of llms into smaller models to improve efficiency. our experiments verify the significance of informativeness in query rewrites and the effectiveness of using llms for generating rewrites. despite the superb performance achieved by our proposed approach, there are multiple future directions that are worthy of exploration. for example, we can train an auxiliary model to decide whether queries generated by prompting llms should be preferred to those generated by models that have been finetuned on human rewrites. we can also combine human rewrites and llm rewrites as labels through a proper weighting strategy to finetune query rewriting models. furthermore, in this work, we have used a fixed set of demonstrations for all test queries. to achieve the best performance, it is essential to find appropriate demonstrations for each specific query. this would be an effective solution to tackle obscure or complicated queries. another future direction can be parameter-efficient fine-tuning (e.g., lora (hu et al., 2021)) of llms with retrieval performance as feedback. in this way, we will aim to optimize the helpfulness of rewritten queries rather than informativeness.