Accelerating Multiple Intent Detection and Slot Filling via Targeted Knowledge Distillation (2023)

Abstract: Recent non-autoregressive Spoken Language Understanding (SLU) models have attracted increasing attention because of their encouraging inference speed. However, most of existing methods (1) suffer from the multi-modality problem since they have little prior knowledge about the reference during inference; (2) fail to achieve a satisfactory inference speed limited by their complex frameworks. To tackle these issues, in this paper, we propose a \textbf{T} argeted \textbf{K} nowledge \textbf{D} istillation \textbf{F} ramework (TKDF) for multi-intent SLU, which utilizes the knowledge distillation method to improve the performance. Specifically, we first train an SLU model as the teacher model, which has higher accuracy while slower inference speed. Then we introduce an evaluator and apply a curriculum learning strategy to select proper targets for the student model. Experiment results on two public multi-intent datasets show that our approach can realize a flexible trade-off between inference speed and accuracy, achieving comparable performance to the state-of-the-art models while speeding up by over 4.5 times. More encouragingly, further analysis shows that distilling only 4% of the original data can help the student model outperform its counterpart trained on the original data by about 14.6% in terms of overall accuracy on MixATIS dataset.

## introduction spoken language understanding (slu) plays a pivotal and indispensable role in task-oriented spoken dialog systems @xcite . it aims to understand the queries of users, which includes two subtasks: intent detection and slot filling @xcite . specifically, intent detection task aims to predict the intent of the given utterance and slot filling task aims to extract the additional information or constraints expressed in the utterance. in real-world scenarios, it is common for an utterance to contain multiple intentions. in response to this challenge, @xcite and @xcite starts addressing the multi-intent slu task. @xcite makes the first attempt to use a multi-task framework to jointly model multiple intent detection and tokens possibility reference city intent: atis_city, atis_airport atis_flight, atis_ground_fare slot: b-toloc.city_name i-toloc.city_name service intent: atis_meal, atis_aircraft atis_flight, atis_ground_fare slot: i-transport_type o angeles intent: atis_airline, atis_airport atis_flight, atis_ground_fare slot: i-airport_name i-city_name for the first problem, a very common solution is to utilize knowledge distillation to first training an autoregressive model in the original training corpus and then applying the greedy outputs of the teacher model as the targets to train the non-autoregressive student model @xcite @xcite . however, there is still a lack of research on applying knowledge distillation to address the multi-modality problem in slu. a very intuitive idea is to directly employ knowledge distillation to non-autoregressive slu models, where the student slu model could learn from the teacher slu model. unfortunately, our preliminary experimental results demonstrate that this method fails to solve multi-modality problem effectively, whose details could be seen in sec.6.3. we argue that the reason for these divergent results is that slu is a classification task, while neural machine translation is a generative task @xcite . considering that it has been verified that directly applying knowledge distillation to the non-autoregressive machine translation models will lead to the significant decrease in terms of prediction accuracy for the low-frequency words @xcite . in slu task, owing to the limited categories of intents and slots, the decrease in terms of prediction accuracy of low-frequency words has a more serious impact on the overall performance. for the second problem, knowledge distillation also seems to be an effective solution. knowledge distillation has been widely used to improve reasoning speed without overly sacrificing performance in various tasks @xcite @xcite . as a result, we decide to advance along this technical path and apply knowledge distillation to choose the teacher model with the higher accuracy and select a student model with the higher inference speed. by this means, the distilled slu model would obtain a higher inference speed while almost maintaining the accuracy. in order to address these above two problems at the same time, we try to improve the past methods instead of directly applying traditional knowledge distillation, thereby mitigating the multi-modality problem and achieving the trade-off between inference speed and accuracy. motivated by the recent success of advanced knowledge distillation in other tasks @xcite @xcite , we propose a targeted knowledge distillation framework (tkdf). specifically, we first train a multi-intent slu model as the teacher slu model with higher accuracy while slower inference speed. we regard its output as the distilled data. then, we train a new slu model with a simpler network on the distilled data as the evaluator and use a curriculum learning strategy to obtain the selected data by replacing the original distilled data with original data dynamically during the training process. finally we train an slu model with the same architecture as the evaluator on the selected data as the student slu model. we also propose a metric to score the outputs, and the evaluator model assesses each utterance in the original training set through scoring the output intents and output slots. the utterances with higher scores are chosen as the targets, which typically contain more slight modality changes compared to the distilled data but show better prediction quality. we believe the selected utterances could be more safely exposed to the student model during training, which is beneficial for the overall performance. we conduct all the experiments on two public benchmark multi-intent slu datasets, mixatis and mixsnips @xcite @xcite and over three slu architectures, co-guiding net @xcite , rela-net @xcite and ssran @xcite . experiment results demonstrate that our method can realize a flexible trade-off between inference speed and accuracy, and further analysis also verifies the advantages of our framework. in summary, the core contributions of this work could be concluded as follows: ## background in general, intent detection and slot filling are two subtasks in slu. given an input utterance @xmath0, x 2 , . . . , x n ), where n is the length of x, multiple intent detection is a multi-label classification task and the reference intent sequence is denoted as ŷi = ŷ(1,i) , ŷ(2,i) , . . . , ŷ(m,i) , where m is the number of intents in x. slot filling is a sequence labeling task and the reference slot sequence is denoted as ŷs = ŷ(1,s) , ŷ(2,s) , . . . , ŷ(n,s) . and we use ŷ to denote the the union of ŷi and ŷs . owing to the correlation between slots and intents, joint models are utilized to jointly optimize these two subtasks. the multi-intent detection objective and the slot filling objective are as follows: ## method as shown in figure 1 , our tkdf could be divided into the following parts. first, we feed the original data into the teacher model with higher accuracy while slower inference speed to obtain the distilled data. then, we utilize an evaluator module to construct the selected data, replacing the distilled data with the original data dynamically. finally, we train the student model on the selected data. the presence of multi-modality problem poses a challenge for the non-autoregressive slu models, because it hinders accurate slot filling for certain tokens, consequently leading to misguided intent detection. furthermore, the wide adoption of tokenlevel intent detection decoders also exacerbates the negative impact of the multi-modality problem on intent detection performance. in contrast, distilled data output by the teacher slu model could simplify the training process of non-autoregressive models by reducing the target complexity. however, relying exclusively on distilled data is not the optimal approach, as it can be vulnerable to the errors made by the teacher model. an intuitive approach is to leverage the strengths of both the original data and the distilled data by combining them. motivated by @xcite ; @xcite , we propose a non-autoregressive evaluator to determine whether the student model should be trained on the original data or the distilled data. the evaluator is first trained on the distilled data. subsequently, we feed the original utterances into the evaluator, which assesses each prediction made on the original data. in cases where the nonautoregressive evaluator fails to produce outputs that closely align with the reference, we substitute the original labels with the corresponding distilled versions. specifically, for the utterance x, we first get the output y including the predicted intents and slots utilizing the non-autoregressive evaluator: then we use the overall accuracy as the metric score to evaluate the original label ŷ. predictions with the higher scores are considered more friendly for the the non-autoregressive student model. the score could be formulated as: inspired by recent success of curriculum learning @xcite @xcite , we apply curriculum learning to improve the performance. as the training progresses, the ratio of original data gradually diminishes. we establish a threshold t , wherein if the score surpasses this threshold, the distilled data will be replaced by the corresponding original data. the specific threshold t k is determined based on the current training step k: ## conclusion in this paper, we propose tkdf, a simple yet effective method to solve the multi-modality problem of non-autoregressive slu. we introduce an evaluator and apply a curriculum learning strategy to select proper targets for the student model. experiments and analysis demonstrate the effectiveness of tkdf, which could achieve a flexible trade-off between inference speed and accuracy.