Attend, Select and Eliminate: Accelerating Multi-turn Response Selection with Dual-attention-based Content Elimination (2023)

Abstract: Although the incorporation of pre-trained language models (PLMs) significantly pushes the research frontier of multi-turn response selection, it brings a new issue of heavy computation costs. To alleviate this problem and make the PLM-based response selection model both effective and efficient, we propose an inference framework together with a post-training strategy that builds upon any pre-trained transformer-based response selection models to accelerate inference by progressively selecting and eliminating unimportant content under the guidance of context-response dual-attention. Specifically, at each transformer layer, we first identify the importance of each word based on context-to-response and response-to-context attention, then select a number of unimportant words to be eliminated following a retention configuration derived from evolutionary search while passing the rest of the representations into deeper layers. To mitigate the training-inference gap posed by content elimination, we introduce a post-training strategy where we use knowledge distillation to force the model with progressively eliminated content to mimic the predictions of the original model with no content elimination. Experiments on three benchmarks indicate that our method can effectively speeds-up SOTA models without much performance degradation and shows a better trade-off between speed and performance than previous methods.

## introduction constructing intelligent dialogue systems has attracted wide attention in the field of natural language processing (nlp) in recent years. there are two approaches widely used for the dialogue we focus on multi-turn response selection in retrieval-based dialogue systems in this paper. recently advances of pre-trained language models @xcite further push the research frontier of this field by providing a much powerful backbone for representation learning @xcite and dialogue-oriented selfsupervised learning @xcite @xcite . although significant performance improvement has been made by these plm-based response selection models, they usually suffer from substantial computational cost and high inference latency due to the growing model size, presenting challenges for their development in resource-limited real-world applications. therefore, there is an urgent need to accelerate plmbased response selection models while maintaining their satisfactory performance. to accelerate plm-based multi-turn response selection, one direct idea is to avoid unnecessary calculation when joint modeling dialogue context and response. through empirical observation, we find that there are many unimportant contents that are either redundant (i.e., repeated by many context turns) or less relevant to the topic, especially in the lengthy dialogue context @xcite . if accurately identified and appropriately eliminated, the removal of the unnecessary calculation on them can bring minimum performance degradation. drawing inspiration from @xcite , we propose an inference framework together with a post-training strategy customized for plm-based multi-turn response selection, where unimportant contents are progressively identified and dropped as the calculation goes from shallow layers to deep. in our framework, we seek to answer three research questions (rqs): (1) how to accurately identify these unimportant contents, (2) how to properly decide the intensity of elimination for these unimportant contents under various computation demands, and (3) how to eliminate unnecessary calculations on those contents at the minimum cost of performance degradation. as the answer to the above questions, we propose an inference framework together with a post-training strategy customized for plm-based multi-turn response selection as illustrated in we evaluate our proposed method on three benchmarks for multi-turn response selection: ubuntu @xcite , douban @xcite and e-commerce @xcite . experimental results show that our proposed method can accelerate the inference of plm-based multiresponse selection models with acceptable performance degradation under various computation constraints, while significantly outperforming previous acceleration methods. we also conduct comprehensive analyses to thoroughly investigate the effectiveness of proposed components. we summarize the contributions of this paper as follows: (1) we propose attend, select and eliminate (ase), an efficient inference framework customized for plm-based multi-turn response selection models that identify and progressively eliminate unimportant contents. (2) we propose a knowledge-distillation-based post-training strategy to mitigate the training-inference gap and decrease the performance degradation caused by content elimination. (3) we conduct comprehensive experiments on three benchmarks to verify the effectiveness of our proposed method and prove its superiority over other acceleration methods. ## related work recently, methods based on pre-trained models are relatively popular, @xcite introduced the next sentence prediction and mask language model tasks in the plms into the conversation corpus, conducted post-domain training, and finally treated the context as a long sequence, and adjusted the model directly by fine-tuning the model. compute context-response match scores. @xcite tries to introduce self-supervised learning tasks to increase the difficulty of model training, and the results show the effectiveness of these works. from the perspective of data augmentation, bert-fp @xcite splits the context into multiple sets of short context-response pairs and introduces a conversational relevance task, which achieves state-of-the-art performance. although the performance of the pre-training model is powerful, it also brings some problems. the expensive computational cost and high inference latency hinder the further implementation of the plms to a certain extent. some works try to alleviate this problem, one of the branches is to reduce the model size, such as distillation @xcite @xcite , structural pruning @xcite al., 2019; gordon et al., 2020; hou et al., 2020) and quantization (zafrir et al., 2019; shen et al., 2020; zhang et al., 2020a; bai et al., 2021), etc. goyal et al. (2020) adopts the attention strategy to select the important tokens with a fixed length configuration, but its speed ratio cannot be selected as needed and once full training can only get a model with a fixed speedup. since existing method goyal et al. ( 2020 ) is mainly evaluated on single-sentence or sentencepair tasks, it not fully suitable for response selection where the model needs to understand the relationship between all the utterances in a dialogue session and learn the interaction of the utterances closely related to the response. therefore, we propose to select and eliminate the token representation based on context-to-response and responseto-context attention (i.e., dual-attention, duala), which make good use of the relationship between context-response. ## methodology we aim to accelerate the inference of plm-based multi-turn response selection models by proposing attend, select and eliminate (ase) that progressively identifies and eliminates unimportant contents to avoid unnecessary calculations. the overall framework is illustrated in figure 1 . there are three crucial questions that need to be answered: ## conclusion in this paper, we propose a new framework of progressively extracting important tokens and eliminating redundant tokens to accelerate inference for multi-turn response selection, which identifies important tokens based on dual-attention of the context and response. the experimental results empirically verify the effectiveness of this method. in the future, we plan to accelerate inference further by combining it with the layer-wise reduction.