Saving Dense Retriever from Shortcut Dependency in Conversational Search (2022)

Abstract: Conversational search (CS) needs a holistic understanding of conversational inputs to retrieve relevant passages. In this paper, we demonstrate the existence of a retrieval shortcut in CS, which causes models to retrieve passages solely relying on partial history while disregarding the latest question. With in-depth analysis, we first show that naively trained dense retrievers heavily exploit the shortcut and hence perform poorly when asked to answer history-independent questions. To build more robust models against shortcut dependency, we explore various hard negative mining strategies. Experimental results show that training with the model-based hard negatives effectively mitigates the dependency on the shortcut, significantly improving dense retrievers on recent CS benchmarks. In particular, our retriever outperforms the previous state-of-the-art model by 11.0 in Recall@10 on QReCC.

## introduction conversational search (cs) is a task of retrieving relevant passages from a large amount of web text given the current question and its conversational history, which consists of previously asked questions and their answers @xcite . unlike open-domain question answering (odqa) taking a single question @xcite , cs assumes a sequence of questions interactively taken from information seekers. hence, the questions need to be understood with the conversational history to find relevant evidence at each turn. to build a retriever that properly makes use of the conversational history, we first analyze a simple dense retriever baseline trained on one of the cs datasets, qrecc @xcite . our analysis shows us the existence of a retrieval shortcut ⋄ work done while interning at naver ai lab 1 the code is available at github.com/naver-ai/cs-shortcut. figure 1 : an example of a retrieval shortcut in conversational search. while we expect the retriever to predict relevant passages by using all conversational inputs up to q 3 (blue solid line), a dense retriever often ignores current turn question q 3 and only exploits previous history, a 2 (red dashed line). we show the shortcut dependency is harmful to robust retrieval. in recent cs datasets, indicating dense retrievers heavily rely on the shortcut and retrieve irrelevant passages. specifically, these shortcuts represent the spurious correlation between the conversational history and the relevant passages, pushing the dense retrievers to ignore current questions. for example, as illustrated in figure 1 , a dense retriever retrieves wrong passages only paying attention to 'russia' and 'world cup' mentioned in the previous history (a 1 , a 2 ) while ignoring the crucial cue 'win the world cup' in the current question q 3 . motivated by our observation, we further test how much the shortcut contributes to the performance of current retrievers. first, we build a simple bm25 baseline, which only takes the previous conversational history as input, but still performs surprisingly well on qrecc. similarly, a dense retriever trained by feeding the conversational history without the current question keeps 70-80% of the original performance. it implies a significant effect of the shortcut dependency on dense retrievers. from our analysis, we find the shortcut is more likely to be learned when the topic of conversation is constant. in other words, performance of the models drops especially when they are asked to answer history-independent questions. to alleviate the overreliance on the shortcut, we explore using hard negative mining strategies, which have been recently proposed in odqa and cs @xcite @xcite . experimental results show the model-based hard negatives make remarkable improvements in various cs benchmarks and are especially helpful to history-independent questions, mitigating the dependency on the shortcut effectively. our retrievers outperform baselines by 11.0 in recall@10 on qrecc. our contributions are summarized in three folds: ## background and related work let x @xmath0, a 1 , ..., a t-1 , q t } is a conversation up to turn t where the q t and a t are the question and answer at turn t. we assume pre-chunked passages collection @xmath1, p 2 , ..., p |c| } for the retrieval. then, the formal objective of conversational search is learning function f : (x t , c) → p t , where the p @xmath2, p 2 , ..., p k } ⊂ c and k ≪ |c|. on the other hand, conversational query rewriting (cqr) is a generative task that rewrites the conversational input x t into a standalone question q ′ t @xcite @xcite @xcite . then, existing retrieval systems such as bm25 take the standalone question q ′ t to find p t at inference time, i.e. f (q ′ t , c) → p t . as a result, the cqr approaches do not require to re-train additional retriever in a conversational manner. however, the approach is limited in triggering information loss and long latency while rewriting the conversation into the standalone question. to overcome the limitations, @xcite ; @xcite attempt to train dense retrievers to directly represent the multi-round questions into a single dense vector. they usually focused on few-shot adaptation or weak supervision utilizing other accessible resources including the standalone questions for hard negative mining. ## retrieval shortcut first, we demonstrate the presence of the shortcut in cs datasets. formally, we define the shortcut as where gold passage p + t can be predicted in top-k predictions even without the current question q t . then, we show how heavily dense retriever relies on the shortcut and how its overall performance is overestimated. ## experiments we hypothesize random in-batch negatives promote the shortcut dependency of the vanilla dpr model because of their easy-to-distinguish nature. thus, we examine hard negative mining as one of the solutions to push the retriever to exploit the shortcut properly. we mainly evaluate it on two cs benchmarks, qrecc and or-quac @xcite ). foot_1 ## conclusion in this work, we show the presence of the shortcut in conversational search, which causes dense retriever often heavily relies on it when trained on inbatch negatives. we find that shortcut dependency hurts the generalization ability of dense retrievers. to save the model from relying on the shortcut, we study various hard negative mining strategies. the retriever trained with hard negatives appropriately takes beneficial information of the shortcut only when needed and achieves the state-of-the-art performance on multiple cs benchmarks.