Explicit Query Rewriting for Conversational Dense Retrieval (2022)

Abstract: In a conversational search scenario, a query might be context-dependent because some words are referred to previous expressions or omitted. Previous works tackle the issue by either reformulating the query into a self-contained query (query rewriting) or learning a contextualized query embedding from the query context (context modelling). In this paper, we propose a model CRDR that can perform query rewriting and context modelling in a unified framework in which the query rewriting’s supervision signals further enhance the context modelling. Instead of generating a new query, CRDR only performs necessary modifications on the original query, which improves both accuracy and efficiency of query rewriting. In the meantime, the query rewriting benefits the context modelling by explicitly highlighting relevant terms in the query context, which improves the quality of the learned contextualized query embedding. To verify the effectiveness of CRDR, we perform comprehensive experiments on TREC CAsT-19 and TREC CAsT-20 datasets, and the results show that our method outperforms all baseline models in terms of both quality of query rewriting and quality of context-aware ranking.

## introduction the recent rising of intelligent assistants triggers the transition of information retrieval systems from ad-hoc search to conversational search @xcite @xcite . searching in a conversational manner provides interactive information exchange between users and the system. such interactive format is usually achieved by multi-turn dialogues, with which the system can understand the user's complex information needs @xcite . in a conversational system, a user often asks follow-up questions about something referred to previous dialogue context @xcite @xcite . therefore, a conversational search query is likely to be context-dependent and not self-contained. the real user intent is hidden underneath not only the current query but the whole query context. figure 1 shows an example of conversational search, from which we can find two most typical linguistic phenomenons that undermine the semantic completeness of a single query in the conversation, namely anaphora and ellipsis @xcite . specifically, anaphora refers to the phenomenon of an expression that depends on an expression in the previous context. for example, in figure 1 , the word "its" in q 2 depends on the expression "the phoenix city" in q 1 . meanwhile, ellipsis refers to the phenomenon of the omission of expressions in the previous context. for example, the complete form of q 3 in figure 1 should be "how about the population of new york?". however, the expression "the population of" is omitted. due to these two key characteristics, a single query in conversational search is inherently incomplete and ambiguous @xcite . there are a few attempts that aim to alleviate such problems. generally, these methods can be divided into two groups. the first group of methods formalize the problem as context-aware query rewriting @xcite @xcite @xcite . most of these methods finetune a pretrained language model (e.g.gpt2 and t5) to generate a new query via an auto-aggressive decoder @xcite . the second group of methods are on the top of dense retrieval models @xcite @xcite @xcite . these methods seek to tackle the query ambiguity by learning a contextualized dense representation from the query context, which can be then used for dense retrieval @xcite . though these dense retrieval methods booster the performances in many information retrieval tasks, we argue that the current dense retrieval methods have two limitations in conversational search: (1) previous conversational dense retrieval models fail to reformulate the original query into a well-formed readable query, which is useful for re-ranking, providing explainability and benefit other conversational scenarios (e.g.query suggestion or query clarification) ; (2) most of the current dense retrieval models are trained on ad-hoc queries @xcite . and we speculate that these models lack the ability to highlight important terms in the conversational query context accurately (will discuss in section 4). the two limitations of current conversational dense retrieval methods undermine the potentiality of a conversational search system. in this paper, we propose a conversational query rwriting method for dense retrieval (crdr). it explores enhancing the context modelling ability of a dense retrieval model with term supervision signals learned during query rewriting. in other words, crdr simultaneously learns a term-enhanced contextualized query embedding for a better ranking and reformulates the original query into a well-formed query. the advantages of crdr are two-fold. first, crdr can generate a well-formed query which is beneficial as mentioned above. second, with the explicitly enhanced term weighting inspired by the query reformulation, the learned conversational query embedding can better represent the user's intent and will lead to better retrieval quality. crdr achieves the goal via two modules, namely the query rewriting module and the dense retrieval module. the query rewriting module treats the task as query modification. specifically, it directly performs token-level modifications (replace or insert) on the original query with relevant terms in the query context. the idea is directly motivated by the linguistic definition of anaphora and ellipsis which are two key characteristics of conversational search (see section 3.3). instead of generating a new query, our method directly use information in the query context to modify the current query. besides, the dense retrieval module learns a contextualized query embedding from the query context in which it considers the relevant terms detected during query writing. during the query rewriting, we measure the relevance between the terms in the query context and the current query. for the terms with high relevant scores, we will dynamically fuse their semantics into the current query embedding. to verify the quality of the reformulated query and contextualized query embedding generated by crdr, we conduct experiments on both traditional inverted index-based retrieval and dense retrieval models. experiment results show that crdr outperforms all baseline models. our contributions can be summarized as: (1) we seek to unify query rewriting with dense retrieval in conversational search and reveal that explicit query rewriting is beneficial to conversational dense retrieval. (2) we propose crdr, which integrates the query rewriting and dense retrieval in a unified framework in which the learned contextualized query embedding is enhanced by the relevant terms detected during query rewriting. (3) crdr can simultaneously learn a term-enhance contextualized query embedding for dense retrieval and generate an explicit query which provides explanability and benefits other tasks. (4) extensive experiments show that our method outperforms the state-of-the-art methods on both traditional inverted index-based retrieval and dense retrieval. ## related work conversational search is considered as one of the most promising searching paradigm in the information retrieval domain @xcite . with the release of trec conversational assistant track (cast), many researchers pay much attention into the task @xcite . the cast tasks contain multi-turn dialouge turns seeking information from the web search engine. the tasks are evaluated by measuring the accuracy of retrieved documents given the dialogue queries. along the direction, previous works either leveraging a query rewriting model to generate a new query or learn a contextualized query embedding to represent the whole query context. query rewriting is a widely applied technique to alleviate query ambiguity in conversational search. it aims to reformulate the original query into a self-contained query with the query context. query rewriting task was first introduced by @xcite . they released the canard dataset deriving from the quac dataset and provides human reformulated queries @xcite . recent trec cast tasks greatly promoted the development of conversational query writing research @xcite . they defined conversational search as a task in which effective passage retrieval requires understanding a query context. follow the cast tasks, @xcite proposed a set of heuristic methods including part-of-speech, dependency parse and co-reference resolution to rewrite the original query. besides, @xcite treated the task as relevance term classification in which each term in the query context is assigned a binary label. they then added the relevant terms at the end of the original query. to generate a wellformed natural language query, many methods treat the task as text generation in which they input the query context into a text generation model and predict a new query @xcite @xcite @xcite . yu et al. (2020) proposed a rulebased query simplification method generating a large number of distantly supervised data to train a text generation model. to increase the generative ability of the text generation model, vakulenko et al. (2021a) proposed a model that considers several individual word distributions during decoding. besides, lin et al. (2021c) proposed a method that considers term importance. a dense retrieval system usually applies a biencoder architecture to encode the query and document into dense vector separately @xcite @xcite . it then computes the query-document relevance with similarity functions (e.g.dot product) and performs retrieval via dense searching algorithms (e.g.approximate nearest neighbor) @xcite . in conversational search, the query embedding is learned from a set of follow-up queries instead of an ad-hoc query. as labelling ground truth query in conversational search is exhausted, it is difficult to train a dense retrieval model dedicated to the conversational system. therefore, @xcite directly concatenated the follow-up queries into a long sequence and fed them to the dense retrieval model finetuned on the canard dataset. through this way, the model can learn a query embedding from the whole query context and then perform dense retrieval. besides, @xcite proposed a few-shot framework to continue training an ad-hoc dense retrieval model. @xcite utilize a curriculum contrastive method to denoise the conversational context for better representations. similarly, they then learn a query embedding which is used in the dense retrieval system. in this paper, we first obtain the well-formed query via an encode-tag-modify framework. then, we use the relevant terms detected in the query context as supervision signals to enhance the robustness of the query encoder. ## discussion in this section, we first conduct a breakdown analysis to explore the pros and cons of sparse retrieval and dense retrieval in conversational search. then, we illustrate how the query rewriting in our crdr can be used to denoise the query context when learning the conversational context representation. in appendix d, we use a case study to explicitly walk through why our crdr performs better than baseline models. ## conclusion in this paper, we explore how to unify query rewriting and dense retrieval in conversational search in one framework. we propose crdr, a model that can simultaneously generate a self-contained query and model the query context into a contextualized query embedding that considers the relevant terms detected during query rewriting. specifically, the query rewriting module directly modifies the original query via an encode-tag-modify framework. the dense retrieval module leverage the relevant terms detected by the query rewriting module as supervision signal to enhance the contextualized query embedding. we conduct comprehensive experiments on the cast-19 and cast-20 datasets. the experiment results show that crdr outperforms all baseline models on both sparse retrieval and dense retrieval.