Two Birds One Stone: Dynamic Ensemble for OOD Intent Classification (2023)

Abstract: Out-of-domain (OOD) intent classification is an active field of natural language understanding, which is of great practical significance for intelligent devices such as the Task-Oriented Dialogue System. It mainly contains two challenges: it requires the model to know what it knows and what it does not know. This paper investigates “overthinking” in the open-world scenario and its impact on OOD intent classification. Inspired by this, we propose a two-birds-one-stone method, which allows the model to decide whether to make a decision on OOD classification early during inference and can ensure accuracy and accelerate inference. At the same time, to adapt to the behavior of dynamic inference, we also propose a training method based on ensemble methods. In addition to bringing certain theoretical insights, we also conduct detailed experiments on three real-world intent datasets. Compared with the previous baselines, our method can not only improve inference speed, but also achieve significant performance improvements.

## introduction with the increasing popularity of intelligent devices such as the task-oriented dialogue system and the increasingly open environment they face, it becomes more and more challenging to accurately understand the intents behind the utterance of the users, on one hand, it needs to ensure the accuracy of in-domain (ind) intents, and on the other hand, it needs to be able to effectively identify ood intent. these days, it has became a de facto standard to take the semantic representation of the last layer of finetuned pre-train models (ptms) for intent understanding @xcite . however, in the closed-world scenario (i.e., figure 1 : illustration of overthinking in the open-world scenario. the model becomes more "confident" (pred entropy decreases with the number of layers.), but the ability to distinguish ind and ood decreases (the difference is getting smaller). the results are obtained with the test set of clinc-small. training and test set come from the same distribution), previous studies have pointed out that ptms would be "overthinking" the semantic features of the sample @xcite , i.e., the representation of the sample may become too complex after reaching the last layer through multiple stacked transformer layers, thus affecting the final decision of the model. naturally, in the open-world scenario (i.e., the condition of the same distribution is not maintained), will the ptms be "overthinking" the difference between ind and ood intents? we want to explore the evolution of the model's ability to distinguish ind and ood through the forward process of two widely used models bert @xcite and albert @xcite . to this end, we first attach extra internal classifiers (ics), which also are referred to as off-ramps, between different internal transformer layers of the model as shown in figure 2 (a) and train these internal classifiers and all transformer layers in two stages as suggested in @xcite . the outputs of these classifiers are used to measure the ability of different layers to distinguish between ind and ood, specifically, by comparing the confidence (calculated by maximum softmax probabil-ity) or uncertainty (calculated by entropy) outputted by each layer on in-and out-of-domain samples. ideally, ind should be given higher confidence (or lower uncertainty) than ood, and a larger difference or gap suggests a stronger ability to distinguish between them. the results are shown in figure 1 . there are two types of curves in figure 1 , where the pred entropy (red) refers to the entropy (average value) of the sample output by the internal layers of the model, which is used to measure the certainty in its output. the curve of "diff" shows the trend of the variation in the difference between the average values of the confidence (blue, and the orange refers to entropy) of all ind samples and ood samples as the number of layers increases. we observe that with the increase of the number of layers, the model becomes more "confident" (lower entropy in output), but the ability to distinguish ind and ood decreases (the difference is getting smaller) after reaching a certain peak. the above experiments show that in the openworld scenario, ptms also would be "overthinking" for the difference between ind and ood. the "overthinking" behavior of the model in different scenarios also inspires us it is unnecessary to forward the last layer to distinguish ind and ood or identify a specific class of ind during inference. to this end, we design an inference strategy to let the passed ics dynamic ensemble and let these ics vote to determine whether the sample is ind or ood and whether they have enough confidence to exit early (section 3.2). furthermore, we introduce a training method that can adapt to the accurate and accelerated behavior in the inference stage. on the one hand, according to the theory of ensemble methods, we can improve the accuracy of ind recognition of the ensembled ics by reducing the redundancy between ics to increase the diversity among ics during the training process. on the other hand, we can reduce the open space risk by regulating the recognition behavior of the adjacent space of each training sample during the training process to improve the ood detection capability. in addition to the theoretical explanation of the reason for introducing the training method, we also carry out detailed verification on three challenging intent datasets. the experiment shows that our method can not only substantially improve the accuracy, but also effectively accelerate inference. we summarize our key insights and contributions as follows: firstly, we explore the "overthinking" phenomenon of the model in the open-world scenario and its impact on the discrimination between ind and ood. we hope that these explorations can inspire follow-up studies. secondly, we design a simple but effective inference strategy to improve the model recognition ability and speed up the model inference. at the same time, we also introduce a novel training method to adapt the inference strategy. thirdly, detailed experiments show that compared with the existing methods, our proposed method can substantially improve the accuracy but also effectively accelerate inference. our work is related to two types of research: ood intent classification and early exiting. ood intent classification is also known as ood detection. such research can be roughly divided into two categories according to whether additional ood samples are introduced (or synthesized) in the training process: supervised (the training set contains additional ood samples) and unsupervised (there are no additional ood samples in the training set). since supervised methods require a large number of time-consuming ood samples and the constructed ood samples cannot cover all ood classes, this work focuses on unsupervised methods. scheirer et al. (2012) articulate open space risk as the optimization objective of the task through establishing a constrained optimization. open-max @xcite models the output of the penultimate layer of the network as a specific distribution (weibull distribution) for ood detection. msp @xcite uses the confidence (maximum softmax probability) of the network to distinguish ind and ood. doc @xcite connects n binary classifiers at the end of the network for ood classification. lcml @xcite introduces margin loss to learn more discriminative semantic features to facilitate ood detection. adb @xcite reduces open space risk by introducing adaptive boundaries for each ind class. @xcite extract image features by virtue of projecting the complexity of the images onto the network layers to construct an ood detector in the domain of computer vision. @xcite use all the intermediate representations of pre-trained transformers to build an ood detector. scl @xcite learns to adapt to downstream detection features by contrastive learning paradigms. knn-cl @xcite further improves the learned semantic features by constraining positive and negative samples in contrastive learning. different from the existing work, our method can dynamically decide whether to exit early without going through the whole model during inference, which speeds up the speed of inference and effectively guarantees the accuracy of recognition. early exiting it is considered an effective way to accelerate the speed of inference of large pretrained language models @xcite @xcite . the mechanism allows samples to exit from some internal layers of the model in advance to accelerate the inference process of the model without losing accuracy as much as possible. however, the existing research on early exiting is based on the closed-world scenario. in this work, based on further elaborating the manifestations of the "overthinking" phenomenon in the open-world scenario, we introduce the early exiting mechanism into the open-world scenario to help improve the model's ability to distinguish ind and ood while accelerating the inference of the model. ## conclusion in this paper, we explore whether the model would be overthinking in the open-world scenario and 10666 demonstrate how to affect the discrimination between ind and ood. on this basis, we propose a two-birds-one-stone method, i.e., during inference, let the model independently choose whether to exit early without going to the last layer, which not only ensures the accuracy of recognition but also accelerates the speed of inference. at the same time, we also introduce a training method that can adapt to the dynamic inference of the model. detailed experiments and analysis show that our method can not only accelerate inference but also establish substantial improvements.