KNN -Contrastive Learning for Out-of-Domain Intent Classification (2022)

Abstract: The Out-of-Domain (OOD) intent classification is a basic and challenging task for dialogue systems. Previous methods commonly restrict the region (in feature space) of In-domain (IND) intent features to be compact or simply-connected implicitly, which assumes no OOD intents reside, to learn discriminative semantic features. Then the distribution of the IND intent features is often assumed to obey a hypothetical distribution (Gaussian mostly) and samples outside this distribution are regarded as OOD samples. In this paper, we start from the nature of OOD intent classification and explore its optimization objective. We further propose a simple yet effective method, named KNN-contrastive learning. Our approach utilizes k-nearest neighbors (KNN) of IND intents to learn discriminative semantic features that are more conducive to OOD detection. Notably, the density-based novelty detection algorithm is so well-grounded in the essence of our method that it is reasonable to use it as the OOD detection algorithm without making any requirements for the feature distribution. Extensive experiments on four public datasets show that our approach can not only enhance the OOD detection performance substantially but also improve the IND intent classification while requiring no restrictions on feature distribution.

## introduction people are getting accustomed to talking with or sending some instructions to task-oriented dialog system in natural language to assist them in fulfilling work. as the environment facing the dialogue systems becomes more open, there are more utterances with unknown or out-of-domain (ood) intents that the dialog system does not know how to handle. as shown in figure 1 , the chatbot encounters an unsupported intent (ood intent) utterance in the third interaction. it is significant to distinguish these utterances for the dialogue system because identifying the intents of the user determines whether subsequent actions can be carried out correctly. to solve this problem, the existing methods roughly can be summarized into two categories according to whether extensive labeled ood intent samples are used during training. the first kind of method (use ood samples during training) is represented by @xcite @xcite which regards ood intent classification as a (n+1)-class classification task that the extra (n + 1) th class represents labeled ood intent. these methods may need additional large and time-consuming labeled out-of-domain samples. moreover, manually constructed ood samples endowed with artificial inductive bias cannot cover all open classes in the actual environment so this kind of method have their limitations. in this paper, we focus on another kind of method which involves two stages, to learn discriminative semantic features and ood detection. how to learn semantic features benefit for ood detection? minimizing intra-class variance and maximizing inter-class variance, whose motivation are to facilitate detection by widening margin between ind and ood intents, have always been regarded as the essence of solving this problem @xcite . then, @xcite ; @xcite ; @xcite ; @xcite impose (or implicitly) gaussian distribution into the distribution of the learned intent features to ood detection. in general previous methods implicitly assume the region of semantic features as compact or simply-connected region in feature space, which means ood intents only exist between different ind classes and not within ind distribution so that tight ind semantic features can be helpful for ood detection. however, as shown in figure 2 (a), the actual location of ood intents in semantic space is not limited, they can appear between ind classes or within ind distribution. we name those ood intents that are distributed between different ind classes as inter ood intents, and those within ind distribution or can be surrounded by a convex hull composed of local ind intent samples as intra ood intents. as shown in figure 2 (b), for inter ood intents, minimizing intra-class variance and maximizing inter-class variance can reduce the risk of being identified ind while this risk may be increased for intra ood intents due to being closer to ind intents. at the same time, we conduct gaussian hypothesis testing on the ind semantic features distribu-tion in the clinc-full train set, which is learned by @xcite . we find only 57% ind classes conform to gaussian distribution, which illustrates the gaussian assumption for ood detection in the previous methods may not be reasonable, see appendix a.1 for results on other datasets and details. to solve these problems, we explicitly define the optimization objective of ood intents classification using open space risk @xcite . compared with the previous methods only considering inter ood intents, we propose a simple yet effective method to consider both intra and inter ood intents. we utilize k-nearest neighbors of ind intent sample as positive samples as shown in figure 2 (c), and obtain more negative samples with the help of queue in moco @xcite to learn discriminative semantic features. we further analyze why our method can better reduce open space risk. intuitively, our method leaves more margin around ood intents, which can ensure we employ the basic density-based method for ood detection without any assumptions about the distribution. we summarize our contributions as follows. firstly, following open space risk, we explicit the optimization objective of ood intent classification to provide a paradigm for solving ood intent classification. secondly, we analyze the limitation of existing methods and propose a novel method to better reduce both empirical risk and open space risk. thirdly, extensive experiments conducted on four challenging datasets show our approach achieves consistent improvements without restrictions on feature distribution. the above methods mainly focus on computer vision and assume (or implicitly) the feature region is compact (simply-connected region), many research works are carried out in natural language processing. doc @xcite builds a multi-class classifier and selects a threshold to reject. further, they reduces the open space risk for rejection by tightening the decision boundaries of sigmoid functions with gaussian fitting. lmcl @xcite learns discriminative deep features with margin loss. @xcite ; @xcite model embeddings with a gaussian mixture distribution to facilitate downstream outlier detection. @xcite al. (2020); zeng et al. (2021b); podolskiy et al. (2021) assume the ind semantic feature distribution as gaussian discriminant analysis (gda) and identify out-of-domain samples by mahalanobis distances. @xcite reduce open space risk by decision boundaries and abd @xcite propose to learn adaptive circular decision boundaries. very recently, @xcite propose a supervised contrastive learning objective to maximize inter-class variance and to minimize intra-class variance. these methods also restrict the feature distribution in the feature learning stage or downstream detection stage and fail to solve out-of-domain classification completely. ## conclusion in this paper, we explicit the optimization objective of ood intent classification. we analyze the limitation of existing methods and propose a simple yet effective method to learn discriminative semantic features. our approach pulls together k-nearest neighbors of ind intents and pushes apart them from different class samples to better reduce both empirical risk and open space risk. extensive experiments conducted on four challenging datasets show our approach achieves consistent improvements without restrictions on feature distribution. li-ming zhan, haowen liang, bo liu, lu fan, xiao-ming wu, and albert y. s. lam. 2021. out-of-scope intent detection with self-supervision and discriminative training. in proceedings of the 59th annual meeting of the association for computational linguistics and the 11th international joint conference on natural language processing, acl/ijcnlp 2021, (volume 1: long papers), virtual event, august 1-6, 2021, pages 3521-3532. association for computational linguistics. hanlei zhang, hua xu, and ting-en lin. 2021. deep open intent classification with adaptive decision boundary. in proceedings of the aaai conference on artificial intelligence, volume 35, pages 14374-14382. yinhe zheng, guanyi chen, and minlie huang. 2020. out-of-domain detection for natural language understanding in dialog systems. ieee/acm transactions on audio, speech, and language processing, 28:1198-1209. a appendix dataset classes rate * clinc-full 150 57% clinc-small 150 36% stackoverflow 20 21%