Contextualized Query Embeddings for Conversational Search (2021)

Abstract: This paper describes a compact and effective model for low-latency passage retrieval in conversational search based on learned dense representations. Prior to our work, the state-of-the-art approach uses a multi-stage pipeline comprising conversational query reformulation and information retrieval modules. Despite its effectiveness, such a pipeline often includes multiple neural models that require long inference times. In addition, independently optimizing each module ignores dependencies among them. To address these shortcomings, we propose to integrate conversational query reformulation directly into a dense retrieval model. To aid in this goal, we create a dataset with pseudo-relevance labels for conversational search to overcome the lack of training data and to explore different training strategies. We demonstrate that our model effectively rewrites conversational queries as dense representations in conversational search and open-domain question answering datasets. Finally, after observing that our model learns to adjust the L2 norm of query token embeddings, we leverage this property for hybrid retrieval and to support error analysis.

## introduction with the growing popularity of virtual assistants (e.g., alexa and siri), information seeking through dialogues has attracted many researchers' attention. to facilitate research on conversational search (convs), @xcite organized the trec conversational assistance track (cast) and defined convs as the task of iteratively retrieving passages in response to user queries in a conversation session. an example conversation in the cast dataset is shown at the top of figure 1 @xcite . there are two main challenges for the task of conversational search: (1) user utterances are often ambiguous when treated as stand-alone queries since omission, coreference, and other related linguistic phenomena are common in natural human dialogues. hence, directly feeding the utterances into ir systems would lead to poor retrieval effectiveness. understanding queries through conversational context is required. (2) there is limited data regarding conversational search for model training. to address the aforementioned challenges, existing papers @xcite @xcite ) take a multi-stage pipeline approach. they train a conversational query reformulation (cqr) model using publicly available datasets @xcite and feed the automatically decontextualized queries to an off-the-shelf ir pipeline @xcite . however, such convs pipelines can be slow (i.e., over 10s per query on gpus). furthermore, this design assumes that the reformulated queries are independent of the downstream ir pipeline, which may not be true. in this paper, we study a low-latency end-to-end approach to convs. specifically, we adopt a biencoder model and incorporate cqr into the query encoder, illustrated in figure 1 @xcite . to overcome the challenge of limited training data, we create a dataset with pseudo-relevance labels to guide the query encoder to rewrite conversational queries in latent space directly. one may consider this approach as throwing conversational queries into a black box since the reformulated queries are represented as dense vectors. however, we find that the fine-tuned contextualized query embeddings (cqe) are easily interpretable. they can be transformed into text for failure analysis and can facilitate densesparse hybrid retrieval. our contributions are summarized as follows: (1) we integrate two tasks in convs, query reformulation and dense passage retrieval, into our dense representation learning framework. due to the lack of human labeled data, we create a dataset with pseudo-relevance labels for model training. we empirically show that our model successfully learns to reformulate conversational queries in a latent representation space. (2) we uncover how cqe learns to reformulate conversational queries in a latent space. based on this finding, we can easily transform cqe into a text (sparse) representation. we demonstrate that the cqe text representation also performs well on sparse retrieval and can further improve cqe retrieval effectiveness using a hybrid of sparse and dense retrieval. the cqe text also helps us understand why the technique fails or succeeds. (3) we show that the query latency of cqe (without re-ranking) is at least an order of magnitude lower than existing multi-stage convs pipelines while yielding competitive retrieval effectiveness. hence, cqe is superior for integration with other models in downstream tasks. (4) we empirically demonstrate its effectiveness in open-domain conversational question answering in a zero-shot setting. ## preliminaries let us define a sequence of conversational queries @xmath0, • • • , q i-1 , q i ) for a topic-oriented session s, where q i stands for the i-th user query (@xmath1. the goal of conversational search is to find the set of relevant passages p + i for the user query q i at each turn, given the conversational context @xmath2. thus, the task can be formulated as the objective: since end-to-end training data for conversational search is extremely limited, a common approach is to factorize f convs θ into a multi-stage pipeline. in a multi-stage pipeline, the components can be tuned with data collected at different stages: ## our approach in this section, we first explain why a bi-encoder design is a good fit for conversational search, and then introduce our contextualized query embeddings (cqe) for conversational search. ann search dense index avgpool cqe w h a t w a s t h e n e o l i t h i c r e v o l u t i o n ? w h e n d i d i t s t a r t a n d e n d ? [ q ] w h y d i d i t s t a r t ? conversational query encoder q 1 <4 q 2 <4 q 3 <4 q 4 <4 q 5 <4 q 6 <4 q 7 <4 q 8 <4 q 9 <4 q 10 <4 q 11 <4 e q 4 θ context query q 1 4 q 2 4 q 3 4 q 4 4 w h a t w a s t h e n e o l i t h i c r e v o l u t i o n ? w h e n d i d i t s t a r t a n d e n d ? [ q ] w h y d i d i t s t a r t ? e @xmath3 ## experimental setup datasets. we conduct experiments on trec cast datasets. trec organized the conversational assistance tracks @xcite , aiming to collect reusable collections for conversational search. the organizers have created relevance judgments (relevance grades 0-4) for each query using assessment pools from participants. in total, there are three datasets available, cast19 (training and eval) and cast20 (eval). 1 the dataset statistics are listed in ## conclusions in this paper, we study how to simplify the multistage pipeline for conversational search and propose to integrate modules for conversational query reformulation (cqr) and dense passage retrieval into our dense representation learning framework. to address the lack of training data for conversational search, we create a dataset with pseudorelevance labels and explore different training strategies on this dataset. experiments demonstrate that our model learns to reformulate conversational queries in a latent space and generates con-textualized query embeddings (cqe) for conversational search. in addition, our analyses provide insight into how cqe learns to rewrite conversational queries in this latent space. finally, we show that there are two main advantages of cqe: first, the effectiveness of cqe is on par with state-of-theart multi-stage pipelines for conversational search, but with much lower query latency. second, cqe serves as a strong dense retriever for open-domain conversational question answering. limitations and future work. our work shows the feasibility of integrating conversational query reformulation and ad hoc retrieval into a bi-encoder dense representation learning framework. however, it is unclear whether the same strategy can be applied to a cross-encoder re-ranker, which, although much slower, still achieves the highest levels of effectiveness. another limitation of our work is that only historical queries are considered as context; nevertheless, in a real conversational scenario, other types of contexts should also be considered, e.g., system responses and conversations between multiple speakers (if present). there is still much to explore around dense representations in these scenarios, which we leave to future work. finally, as shown in @xcite , incorporating sparse retrieval signals into the training of dense retrieval improves dense-sparse fusion effectiveness. we suspect that there is more to be gained from better fusion of dense and sparse results for conversational search.