CORE : Cooperative Training of Retriever-Reranker for Effective Dialogue Response Selection (2023)

Abstract: Establishing retrieval-based dialogue systems that can select appropriate responses from the pre-built index has gained increasing attention. Recent common practice is to construct a two-stage pipeline with a fast retriever (e.g., bi-encoder) for first-stage recall followed by a smart response reranker (e.g., cross-encoder) for precise ranking. However, existing studies either optimize the retriever and reranker in independent ways, or distill the knowledge from a pre-trained reranker into the retriever in an asynchronous way, leading to sub-optimal performance of both modules. Thus, an open question remains about how to train them for a better combination of the best of both worlds. To this end, we present a cooperative training of the response retriever and the reranker whose parameters are dynamically optimized by the ground-truth labels as well as list-wise supervision signals from each other. As a result, the two modules can learn from each other and evolve together throughout the training. Experimental results on two benchmarks demonstrate the superiority of our method.

## introduction the development of a smart human-computer conversation system has been a longstanding objective in the field of artificial intelligence. recent years have seen an increase in interest in constructing dialogue systems through data-driven approaches, leveraging advancements in deep learning techniques @xcite . with the help of information retrieval (ir) techniques to select an appropriate response from a pre-built index @xcite , or text generation techniques to synthesize a * corresponding author. retrieval-based dialogue systems usually follow the retrieval-reranking paradigm @xcite , i.e., two-stage retrieval model, where the model first retrieves a bundle of response candidates from a pre-built index by a fast retriever and then selects an appropriate one with a more sophisticated yet costly response reranker. specifically, as for the retriever, early methods based on hand-crafted features @xcite (e.g., bm25) for fast retrieval, however suffering from vocabulary mismatch problem, especially in context-to-response retrieval. a recent trend is resorting to deep neural model to represent text as dense embeddings in latent semantic space, which is known as siamese encoder or bi-encoder @xcite @xcite @xcite . attributed to the separate encoding paradigm, it can calculate the embeddings of large-scale response candidates to pre-build vector retrieval index, benefiting from high efficiency during online inference. however, it sacrifices fine-grained interactions between a context and the response candidates but only remains sentence-level metric learning, leading to inferior ranking performance. as a remedy, a common practice is to apply a costly yet effective reranker to the retrieved candidates for more precise response selection @xcite @xcite . this is usually achieved by a cross-encoder operating on the text concatenation of the context and each response for its reranking score. in existing two-stage retrieval models from ir tasks, the retriever and reranker are usually optimized in independent ways @xcite @xcite , or distill the knowledge from a pre-trained reranker into the retriever in an asynchronous fashion @xcite . while the knowledge distillation from the reranker can improve the performance of the retriever, the reranker's parameters are usually frozen so it cannot learn from the feedback from the retriever for a positive loop -the feedback can be (i) the retriever built upon a heterogeneous structure can offer a distinct view to regularize the reranker, and (ii) the reranker conversely can provide more effective supervision to make the retriever more generalizable. however, how to train these two modules in a joint way is still an open question. to this end, we propose to unify the training process for both the retriever and the reranker for their mutual benefits in a retrieval-based dialogue system. specifically, we introduce a cooperative training of the retriever and the response reranker (named core) whose parameters are dynamically optimized by the ground-truth labels as well as listwise supervision signals from each other, which enables two models to learn from each other throughout the training process. by combining the fast dense retriever and smart response reranker with a unified architecture and a cooperative training manner, our framework achieves impressive performance while demonstrating acceptable efficiency. we conduct experiments on two benchmarks, i.e., ubuntu dialogue corpus @xcite and the response selection track of dialog system technology challenge 7 (abbr. dstc7) @xcite , where the model is required to select the best response from a candidate pool. evaluation results indicate our model is significantly better than existing models on the benchmarks, and the cooperative training brings consistent improvements over both the retriever and reranker. to sum up, our main contributions are three-fold: ## related works retrieval-based dialogues. in the past, retrievalbased dialogue systems focused on single-turn response selection using message-response pairs as inputs for matching models, as demonstrated in early studies such as @xcite @xcite . however, more recent attention has been given to multi-turn response selection using context-response matching. this includes methods such as dual-lstm @xcite , multi-view matching model @xcite , deep attention matching network (dam) @xcite , and multi-hop selector network (msn) @xcite . with the success of pre-trained language models @xcite in various nlp tasks, researchers have started to apply them to response selection. for instance, vig and ramea (2019) used bert to represent utterance-response pairs and fused these representations to calculate the matching score. similarly, whang et al. (2020) treated context as a long sequence and conducted contextresponse matching with bert. furthermore, gu et al. (2020) incorporated speaker-aware embeddings into bert to enhance the ability of multiturn context understanding. efficient information retrieval. existing information retrieval models @xcite @xcite usually adopt a pipeline method where an efficient first-stage retriever retrieves a small set of candidates from the entire corpus, and then a powerful but slow second-stage ranker reranks them. however, most of the models rely on traditional lexical-based methods (such as bm25) to perform the first stage of retrieval and the ranking models of different stages are learned separately. recently, as a promising approach, dense retrieval (dr) has been widely used for ad-hoc retrieval @xcite @xcite and open-domain question answering @xcite @xcite because it is as fast as traditional methods and can achieve impressive performance. in retrieval-based dialogue, @xcite presents the poly-encoder, an architecture with an additional learned attention mechanism that represents more global features from which to perform self-attention, resulting in performance gains over bi-encoders and large speed gains over plm-based models. besides, @xcite introduce convert which is a compact dual-encoder pretraining architecture for neural response selection. @xcite utilize knowledge distillation to compress the cross-encoder network as a teacher model into the student bi-encoder model. joint training of bi-and cross-encoder. few works in passage/document retrieval have been proposed to train the bi-and cross-encoder jointly but stand with different motivations or/and targets. for example, ar2 @xcite proposes an adversarial method, where it regards the bi-encoder as a retrieval-based generator for the hard negatives to fool the discriminator built upon a crossencoder; rocketqav2 @xcite passes the ground-truth labels to cross encoder and learns bi encoder based solely on the ranking scores from the cross-encoders. to the best of our knowledge, this paper makes the first attempt to combine the efficient dense retriever and smart response selector for building an effective response retrieval system. besides, different from traditional single-directional distillation (from reranker to retriever) @xcite in dialogue, we jointly learn response retriever and selector with a cooperative training framework, where reranker also receives weak listwise supervision signals provided by the retriever. our training schema is similar to the idea of mutual learning @xcite and enables mutual knowledge transfer in a synchronous way. evaluation results also reveal that the retriever and the reranker can co-improve and our full-ranking performance is better than existing distillation methods. ## methodology problem formalization given a data set @xmath0, c, r) z } n @xmath1, ..., u nc } represents a n c turns of conversation context with u i the ith turn, r is a response candidate, and @xmath2, 1} denotes a label with @xmath3. the goal of the task of response selection is to build a matching model ϕ(•, •) from d. for any input context c and a candidate response r, ϕ(c, r) gives a score that reflects the matching degree between c and r. ac-cording to ϕ(c, r), one can rank a set of response candidates for response selection. in particular, the definition of ϕ(•, •) can be a single-stage model or a two-stage model. overall framework retrieval models re-use existing human conversations and select a proper response from a group of candidates for new user input. our method is designed within the retrievalthen-rerank paradigm. specifically, given a message or a conversation context (i.e., a message with several previous turns as conversation history), we use a fast dense retrieval method based on a pretrained bi-encoder architecture as the retriever. in the response re-ranking stage, we employ a more powerful architecture (such as a cross-encoder) to re-rank a small number of the most promising candidates provided by the fast retrieval model. to further improve the effectiveness of the overall system, we introduce a cooperative training of the retriever and the response reranker whose parameters are dynamically optimized by the ground-truth labels and list-wise supervision signals provided by each other, which enables two modules to evolve together and learn from each other throughout the joint training. ## experiments we evaluate the proposed method on two benchmark datasets for both single-state and two-stage multi-turn response selection tasks. ## conclusion in this paper, to build an effective retrieval-based dialogue system, we explore combining the fast dense retriever and the smart response reranker based on plms with better cooperative training schema. specifically, we propose optimizing the response retriever and the reranker at the same time via cooperative training loss, which enables the two modules to learn from each other throughout the training process. experimental results on two benchmarks demonstrate the effectiveness of our proposed framework.