A dapter D istillation: Non-Destructive Task Composition with Knowledge Distillation (2023)

Abstract: Leveraging knowledge from multiple tasks through introducing a small number of task specific parameters into each transformer layer, also known as adapters, receives much attention recently. However, adding an extra fusion layer to implement knowledge composition not only increases the inference time but also is non-scalable for some applications. To avoid these issues, we propose a two-stage knowledge distillation algorithm called AdapterDistillation. In the first stage, we extract task specific knowledge by using local data to train a student adapter. In the second stage, we distill the knowledge from the existing teacher adapters into the student adapter to help its inference. Extensive experiments on frequently asked question retrieval in task-oriented dialog systems validate the efficiency of AdapterDistillation. We show that AdapterDistillation outperforms existing algorithms in terms of accuracy, resource consumption and inference time.

## introduction recently task-oriented dialogue systems have found extensive applications in diverse business domains @xcite @xcite . owing to the idiosyncratic features of these domains, custom dialogue systems are often required. nonetheless, the fundamental functions and architectures underlying these systems typically exhibit noteworthy similarities. hence, the adoption of a platform-based strategy for accommodating task-oriented dialogue systems across multiple domains has emerged as a promising and effective approach. one popular method is called multi-task learning (mtl), which aims to train multiple tasks simultaneously based on the shared representation of all tasks as shown in figure 1 , resulting in relatively good performance on each task @xcite and @xcite . however, new tenants often register on the platform in a streaming manner. therefore, predictions for the existing tenants in mtl would be compromised when new tenants are added to the platform since retraining often occurs at that time. ## problem definition as we know, training adapters for n tasks in parallel might not be practical since tenants often register on the platform in a streaming manner. based on this fact, the time for the j-th task registered on the platform is assumed to be earlier than that for the i-th task, that is t @xmath0, when @xmath1, t 2 , ..., t n }. throughout the paper, we have the following settings which are typically true in practice: based on the above setting, in this paper we are aimed at maximizing the transfer of knowledge from the existing tasks to the current new task without impacting the existing tasks, which is more suitable for a practical scenario where each task is registered on the platform in a streaming manner. ## adapterdistillation adapterfusion allows sharing of information between different tasks through an extra fusion layer at the cost of longer inference time and larger fusion layer @xcite . however, as a new task is registered on the platform, the existing tasks will be impacted in. in order to mitigate this, we propose adapterdistillation to allow sharing of information from the existing tasks to the new one while avoiding the impact of the existing tasks without increasing inference time. ## experiments to validate the effectiveness of adapterdistillation in terms of accuracy, resource consumption and inference time, its performance is evaluated through a practical scenario where frequently asked question (faq) retrieval is considered in task-oriented dialog systems. ## conclusions and future work we proposed a novel and plug-and-play multiadapter knowledge distillation algorithm called adapterdistillation to implement the sharing of information between different tasks. specifically, our proposed algorithm consisted of two stages of training. in the first stage of training, taskspecific knowledge was extracted by training a student adapter using local data. then in the second stage, knowledge from the existing teacher adapters was distillated into this student adapter through optimizing the distillation loss. note that adap-terdistillation only employed the trained student adapter to implement inference, which resulted in fast inference time and low resource consumption. our proposed adapterdistillation algorithm outperformed existing algorithms in terms of accuracy, resource consumption and inference time, meeting a practical scenario where numerous tenants accessing the platform in a streaming manner. in the future, plugging more advanced adapter structures into adapterdistillation is an interesting direction to explore.