CONQRR : Conversational Query Rewriting for Retrieval with Reinforcement Learning (2022)

Abstract: Compared to standard retrieval tasks, passage retrieval for conversational question answering (CQA) poses new challenges in understanding the current user question, as each question needs to be interpreted within the dialogue context. Moreover, it can be expensive to re-train well-established retrievers such as search engines that are originally developed for non-conversational queries. To facilitate their use, we develop a query rewriting model CONQRR that rewrites a conversational question in the context into a standalone question. It is trained with a novel reward function to directly optimize towards retrieval using reinforcement learning and can be adapted to any off-the-shelf retriever. CONQRR achieves state-of-the-art results on a recent open-domain CQA dataset containing conversations from three different sources, and is effective for two different off-the-shelf retrievers. Our extensive analysis also shows the robustness of CONQRR to out-of-domain dialogues as well as to zero query rewriting supervision.

## introduction passage retrieval in an open-domain conversational question answering (cqa) system @xcite , compared to standard retrieval tasks @xcite , poses new challenges of understanding user questions within the dialogue context. most existing conversational retrieval models @xcite @xcite rely on training specific retrievers like dual encoders @xcite . however, re-training well-established retrievers for conversational queries can be expensive or even infeasible due to their complicated system designs (e.g., those used in search engines). moreover, the preference and availability of such off-the-shelf retrievers can vary depending on the end users. the task of question-in-context rewriting or query rewriting (qr) in a conversation @xcite is to convert a context-dependent question into a self-contained question. it enables the use of any off-the-shelf retriever ( therefore, in this paper, we focus on query rewriting for the task of conversational passage retrieval in a cqa dialogue with any off-the-shelf retrieval system that can only be used as a black box. specifically, we seek to build a qr model that rewrites a user query into the input of the retriever, in such a way that optimizes for passage retrieval performance. figure 1 shows an example of our task, where given an off-the-shelf retriever, the agent rewrites the current user query "who won?" into a more effective query for retrieval. recent work that leverages qr for conversational passage retrieval @xcite collects human-rewritten queries to train a supervised qr model. however, humans are usually instructed to rewrite conversational queries to be unambiguous to a human outside the dialogue context, which does not necessarily align with the goal in our task-to optimize the retrieval perfor-mance. we conduct comprehensive experiments in section 4.3 to confirm these human rewrites indeed sometimes omit information from the dialogue context that is useful to the retriever. this limitation of human query rewrites impacts supervised training. in addition, prior supervised qr models are agnostic to downstream retrievers as they are separately trained before their predicted rewrites being used for retrieval during inference. we propose a reinforcement learning (rl)based model conqrr (conversational query rewriting for retrieval). it directly optimizes the rewritten query towards retrieval performance, using only weak supervision from retrieval. we adopt a novel reward function that computes an approximate but effective retrieval performance metric on in-batch passages at each training step. our reward function does not assume any specific retriever model design, and is generic enough for conqrr to adapt to any off-the-shelf retriever. we show conqrr outperforms existing qr models on a recent large-scale open-domain cqa dataset qrecc @xcite by over 12% and 14% for bm25 and a neural dual encoder retriever model @xcite ) respectively, averaging over three retrieval metrics. we observe the performance boost on all three qrecc subsets from different conversation sources, including one that only appears in the test set (i.e., out-of-domain). to conclude, our contributions are as follows. 1) we introduce conqrr as the first rl-based qr model that can be adapted to and optimized towards any off-the-shelf retriever for conversational retrieval. 2) we demonstrate that conqrr achieves state-of-the-art results with off-the-shelf retrievers on qrecc with conversations from three sources, and is effective for two retrievers including bm25 and a dual encoder model. 3) our analysis shows conqrr trained with no human rewrite supervision provides better retrieval results than strong baselines trained with full supervision, and is robust to out-of-domain dialogues, topic shifts and long dialogue contexts. 4) we conduct a novel quantitative study to analyze the limitations and utility of human rewrites in retrieval performance, which are largely unexplored in prior work. ## approach problem definition we focus on the task of query rewriting (qr) for conversational passage retrieval in a cqa dialogue, with an off-the-shelf retriever. the task inputs include a dialogue context x consisting of a sequence of previous utterances (u 1 , u 2 , . . . , u n-1 ), the current user question u n , a passage corpus p and an off-the-shelf retriever r. foot_0 r cannot be fine-tuned but returns a ranked list of top-k passages when given a query string and a passage corpus, and no other assumption about the model architecture of r can be made. the task aims to rewrite x into a query q such that r can take q as the input query to retrieve passages relevant to x from p . specifically, a passage p is relevant to x if p provides enough information to answer u n in the context of (u 1 , u 2 , . . . , u n-1 ). in this section, we first describe a supervised qr model based on t5 (t5qr) @xcite ) that applies a generic seq2seq training objective with qr labels (section 3.1). then we introduce our rl-based framework conqrr (conversational query rewriting for retrieval) that trains a qr model to optimize towards retrieval and is adaptable to any given off-the-shelf retriever, with weak retrieval supervision (section 3.2).