Contextual Fine-to-Coarse Distillation for Coarse-grained Response Selection in Open-Domain Conversations (2022)

Abstract: We study the problem of coarse-grained response selection in retrieval-based dialogue systems. The problem is equally important with fine-grained response selection, but is less explored in existing literature. In this paper, we propose a Contextual Fine-to-Coarse (CFC) distilled model for coarse-grained response selection in open-domain conversations. In our CFC model, dense representations of query, candidate contexts and responses is learned based on the multi-tower architecture using contextual matching, and richer knowledge learned from the one-tower architecture (fine-grained) is distilled into the multi-tower architecture (coarse-grained) to enhance the performance of the retriever. To evaluate the performance of the proposed model, we construct two new datasets based on the Reddit comments dump and Twitter corpus. Extensive experimental results on the two datasets show that the proposed method achieves huge improvement over all evaluation metrics compared with traditional baseline methods.

## introduction given utterances of a query, the retrieval-based dialogue (rbd) system aims to search for the most relevant response from a set of historical records of conversations @xcite @xcite . a complete rbd system usually contain two stages: coarse-grained response selection (rs) and fine-grained response selection @xcite . as shown in figure 1 , in coarse-grained rs stage, the retriever identifies a much smaller list of candidates (usually dozens) from large-scale candidate database (up to millions or more), then the ranker in fine-grained rs stage selects the best response from the retrieved candidate list. recent studies @xcite @xcite pay more attention on fine-grained rs and various complex models are proposed to compute the similarities between the query and candidates for response selection. although promising improvements have been reported, the performance of fine-grained stage is inevitably limited by the quality of the candidate list constructed. therefore, a high-quality coarsegrained rs module is crucial, which is less explored in existing literature @xcite . in this paper, we focus on the task of coarsegrained response selection, i.e., dialogue response retrieval. there are two major challenges. first, different from general text matching tasks such as adhoc retrieval @xcite ) or question answering (qa) retrieval @xcite , keywords overlapping between context and response in dialogue are potentially rare, such as when a topic transition @xcite occurs in response. this makes it difficult to directly match the query with candidate responses. second, compared with fine-grained rs, coarse-grained rs deals with much larger number of candidates. therefore, it is impractical to apply complex matching model that jointly process query and response for the similarity computation like in fine-grained rs, due to the retrieval latency (traverse millions of candidates on-line). instead, the efficient bm25 system @xcite based on sparse representations is the mainstream algorithm in coarsegrained text matching. to mitigate the above mentioned two problems, we propose a contextual fine-to-coarse (cfc) distilled model for coarse-grained rs. instead of matching query with response directly, we propose a novel task of query-to-context matching in coarsegrained retrieval, i.e. contextual matching. given a query, it is matched with candidate contexts to find most similar ones, and the corresponding responses are returned as the retrieved result. in this case, the potential richer keywords in the contexts can be utilized. to take the advantage of complex model and keep the computation cost acceptable, we distillate the knowledge learned from fine-grained rs into coarse-grained rs while maintaining the original architecture. for the evaluation, there is no existing dataset that can be used to evaluate our model in the setting of contextual matching, because it needs to match context with context during training, while positive pairs of context-context is not naturally available like context-response pairs. therefore, we construct two datasets based on reddit comment dump and twitter corpus. extensive experimental results show that our proposed model greatly improve the retrieval recall rate and the perplexity and relevance of the retrieved responses on both datasets. the main contributions of this paper are threefold: 1) we explore the problem of coarse ## method in coarse-grained response selection, there is a fixed candidate database containing a large number of context-response pairs. formally, given a query, i.e., a new context, the goal is to retrieve top-k most suitable responses for the query from the candidate database. we propose a contextual fine-to-coarse distillation framework for the task of coarse-grained rs. first, we formulate the problem as a task of contextual matching, i.e., match query with context instead response; second, we utilize a multi-tower architecture to deal with the similarity computation of query and candidates in contextual matching; third, we utilize knowledge distillation to leverage the deep interaction between query and response learned in one-tower architecture. ## datasets construction to evaluate the performance of the proposed model, we construct two new datasets based on the reddit comments dump @xcite and twitter corpus 2 . we create a training set, a multi-contexts (mc) test set and a candidate database for reddit and twitter respectively. for reddit, we create an additional single-context (sc) test set. the motivation for these settings is explained in § 5.3. the size of our candidate database is one million in twitter and ten million in reddit respectively, which is very challenging for response retrieval. ## experiments we conduct extensive experiments on the constructed datasets. in this section, we present experimental settings, evaluation metrics, model performance, human evaluation, etc. to demonstrate the effectiveness of the proposed models. ## conclusion in this paper, we propose a contextual fine-to-coarse (cfc) distilled model. in cfc model, we adopt matching on both query-response and querycontext. considering the retrieval latency, we use multi-tower architecture to learn the dense representations of queries, responses and corresponding contexts. to further enhance the performance of the retriever, we distill the knowledge learned by the one-tower architecture (fine-grained) into the multi-tower architecture (coarse-grained). we construct two new datasets based on reddit comment dump and twitter corpus, and extensive experimental results demonstrate the effectiveness and potential of our proposed model. in the future work, we will further explore how the enhancement of coarse-grained rs can help fine-grained rs.