Distilling the Knowledge of Large-scale Generative Models into Retrieval Models for Efficient Open-domain Conversation (2021)

Abstract: Despite the remarkable performance of large-scale generative models in open-domain conversation, they are known to be less practical for building real-time conversation systems due to high latency. On the other hand, retrieval models could return responses with much lower latency but show inferior performance to the large-scale generative models since the conversation quality is bounded by the pre-defined response set. To take advantage of both approaches, we propose a new training method called G2R (Generative-to-Retrieval distillation) that preserves the efficiency of a retrieval model while leveraging the conversational ability of a large-scale generative model by infusing the knowledge of the generative model into the retrieval model. G2R consists of two distinct techniques of distillation: the data-level G2R augments the dialogue dataset with additional responses generated by the large-scale generative model, and the model-level G2R transfers the response quality score assessed by the generative model to the score of the retrieval model by the knowledge distillation loss. Through extensive experiments including human evaluation, we demonstrate that our retrieval-based conversation system trained with G2R shows a substantially improved performance compared to the baseline retrieval model while showing significantly lower inference latency than the large-scale generative models.

## introduction recently, generative models have shown great success in open-domain conversation along with the development of large-scale language models, yielding fluent and informative responses @xcite @xcite . however, generative models suffer from the challenges of latency and computational resources for meanwhile, retrieval models such as bi-encoder and poly-encoder @xcite is able to build efficient open-domain conversation systems by pre-defining the response set and searching the most relevant response to the given context from the response set. in addition, a bi-encoder dramatically reduces the latency when adopting efficient maximum inner product search (mips) libraries, such as faiss @xcite and scann @xcite . despite the outstanding efficiency, retrieval models have shown some lack of conversational ability compared to generative models. retrieval models are known to return an erroneous response when the pre-defined response set does not contain the proper response to the given context, while generative models deal with these cases more flexibly @xcite . exemplar-based generative models @xcite @xcite try to mitigate this problem by combining the advantages of the two approaches, whereas the inherent inefficiency of the generative models remains since exemplar-based generative models employ a generative model for response generation. to make an efficient yet fluent open-domain conversation system, which is mandatory for realworld applications, we propose a novel training method for retrieval models called generative-to-retrieval distillation (g2r). g2r enables retrieval models to leverage the knowledge of large-scale generative models in both data-level and modellevel. first, data-level g2r augments the original dialogue dataset with the responses produced by a large-scale generative model using contexts in the original dialogue dataset. then, the produced responses are also added to the pre-defined response set. the augmented dialogue dataset and response set are utilized for training a retrieval model at the training phase and for returning responses at the inference phase, respectively. although data-level g2r enables retrieval model to utilize high-quality responses generated by the large-scale generative model, it does not transfer the fine-grained knowledge from the generative model about the quality of individual responses. model-level g2r resolves this limitation by transferring the response quality scores assessed by the large-scale teacher generative model into the scores of the student retrieval model. this method induces the retrieval model to select a better response in terms of the response quality. we empirically demonstrate that a retrievalbased conversation system, which consists of the g2r-applied retrieval model and a mips library, shows a substantial conversational ability while showing fast inference speed, as shown in figure 1 . for instance, our retrieval-based conversation system shows about a 20x speedup compared to the blender model (90m parameters) while exhibiting a comparable human evaluation result on conversational ability. ## conclusion we present g2r, a novel training scheme of retrieval model for open-domain conversation by distilling the knowledge of large-scale generative models in both data-level and model-level. g2r enables retrieval models to build a highly efficient conversation system that exhibits a substantial level of con-versational ability. we believe that our work will serve as a stepping stone for creating an efficient and real-time open-domain conversation system.