A Closer Look at Few-Shot Out-of-Distribution Intent Detection (2022)

Abstract: We consider few-shot out-of-distribution (OOD) intent detection, a practical and important problem for the development of task-oriented dialogue systems. Despite its importance, this problem is seldom studied in the literature, let alone examined in a systematic way. In this work, we take a closer look at this problem and identify key issues for research. In our pilot study, we reveal the reason why existing OOD intent detection methods are not adequate in dealing with this problem. Based on the observation, we propose a promising approach to tackle this problem based on latent representation generation and self-supervision. Comprehensive experiments on three real-world intent detection benchmark datasets demonstrate the high effectiveness of our proposed approach and its great potential in improving state-of-the-art methods for few-shot OOD intent detection.

## introduction intent detection is an important component of taskoriented dialogue system, which aims at accurately identifying the intent behind user utterances. out-of-distribution (ood) intent detection aims to solve a (k + 1)-way classification problem with k in-distribution (id) intent classes and an additional ood class representing malformed or unsupported queries. in practice, ood intent detection is often performed in data-scarcity scenarios, e.g., at the early development stage of a dialogue system when labeled data is not sufficient, or for dialogue systems developed for minority language users where it is difficult to find suitable annotators. despite its practical importance, few-shot ood intent detection is a highly challenging problem, which is seldom studied in the literature and has not been investigated in a systematic way. recent advances in ood intent detection @xcite @xcite commonly assume that there are adequate id examples available for training, without considering the few-shot scenario. to our best knowledge, the only work on this topic is by @xcite , who try to tackle few-shot ood intent detection via transfer learning by fine-tuning roberta @xcite on large-scale natural language inference datasets. in this work, we take a closer look at few-shot ood intent detection and consider a strict setting, where only few-shot in-distribution labeled examples are available during training and no external resources can be exploited, since the requirement of additional resources hinders the applicability of the model. under this simplified yet more challenging setting, state-of-the-art ood intent detection algorithms fail to achieve acceptable performance. in figure 1 , we illustrate the key challenge for few-shot ood detection. as shown in figure 1 , since id classes are under-represented by few-shot id examples, a model based on density estimation @xcite or (k + 1)-way discriminative training @xcite tends to learn a conservative decision boundary and hence there are large margins between the real and learned decision boundaries. real id examples situate in the margins will be inaccurately assigned to the ood class, leading to poor performance. therefore, the key for few-shot ood intent detection is to improve the model performance on id examples. to address this issue, we propose to enrich the training set to improve the representativeness of id intent classes and provide more useful learning signals. we explore the feasibility of generating synthetic id examples in a self-supervised manner. in particular, we train a denoising autoencoder (dae) @xcite in the latent representation space only using the few labeled id examples. the trained decoder of dae is then used to efficiently sample synthetic id examples. with the enlarged training set, we follow @xcite to train a (k + 1)-way classifier by simulating ood examples with the enlarged training set. our contributions are summarized as follows: out-of-distribution intent detection. out-ofdistribution (ood) intent detection (or out-ofdomain intent detection) has attracted much attention in research communities, due to its significant importance to the robustness of dialogue systems. the primary challenge of this task is that there is no labeled ood example available for training and validation. as such, the majority of ood intent detection algorithms relies on manually selecting an appropriate threshold. the first line of works @xcite @xcite uses some statistic as the confidence score of whether an example is ood or not. @xcite pointed out that the negative probability outputted by the softmax function can be a good confidence metric for ood detection. @xcite defined a binary classification task for every in-domain class and used the maximum probability among all these binary classifiers as the confidence score. @xcite developed an adversarial training strategy inspired by gan for ood intent detection. the discriminator in gan was trained to assign lower scores to ood examples. @xcite employed an autoencoder trained on in-domain examples and used the reconstruction score as the ood indicator. however, all these methods require manual effort in selecting a proper threshold for ood discrimination. the second line of works @xcite @xcite proposes to learn decision boundaries for ood examples under some assumption of data distribtuion, e.g., mixture of gaussians. ood examples are assumed to lie in the low-density areas of utterance distribution. @xcite proposed to model the in-domain examples by a mixture of gaussians distribution and select a margin to constrain the variance of each in-domain gaussian component. @xcite also made the mixture of gaussian assumption on in-domain data distribution but proposed to automatically learn the variance of the gaussian components. different from previous methods, a recent work by @xcite proposed to directly learn a (k + 1)-way classifier in an end-to-end manner. they created ood learning signals during training by leveraging external data or constructing simulated ood examples with self-supervised information. few-shot ood intent detection. few-shot ood intent detection considers ood intent detection in low-resource scenarios. it aims at developing a reliable ood detector with only a few examples per each in-distribution class. undoubtedly, this is a highly challenging task given that few-shot intent detection is already a big challenge @xcite . at this point, this task is under-explored and has never been investigated in a strictly low-resourced setting. the most related work is dnnc proposed in @xcite , which tries to mitigate the data-scarcity problem by fine-tuning roberta on external large natural language inference datasets. in this paper, however, we consider using the few-shot labeled examples as the only training resource. ## problem statement and pilot study out-of-distribution (ood) intent detection aims at improving the robustness of a dialogue system with respect to utterances with unknown (or unsupported) intents. the key challenge of ood detection is that real ood samples are inaccessible during training and validation. given an in-distribution (id) set of k known classes, y @xmath0, the ood detection task considers another special ood class y ood to represent any malformed or unsupported utterances. hence, given the input space x × y, the goal of ood intent detection is to learn a (k + 1)-way classifier f ϕ (•) : @xmath1: few-shot ood intent detection is a more challenging setting with the assumption that there are only a few labeled in-distribution (id) examples available during training. in this paper, we consider a strict but practical setting by assuming that there are no additional resources (e.g., labeled or unlabeled auxiliary datasets) available to aid the training of the classifier f ϕ (•) or during fine-tuning pre-trained language models. typically, for each id class in {y k } k @xmath2, there are only ∼ 5 or ∼ 10 labeled examples per class. pilot study. to illustrate the challenges of fewshot ood intent detection, we conduct a pilot study on a commonly used ood intent detection dataset clinc150 @xcite using two recent state-of-the-art approaches @xcite class are used for training. the results are summarized in the primary challenge identified from this pilot experiment for few-shot ood intent detection is then how to improve the performance on indistribution classes and achieve a good balance in performance between id and ood classes. ## experiments to evaluate our proposed method for few-shot outof-distribution (ood) intent detection, we conduct extensive experiments on three real-world benchmark datasets. by comparing with state-of-the-art ood intent detection methods, we find that our method can outperform these baselines by a large margin, especially in extreme few-shot scenarios. moreover, our approach yields a more consistent performance at different few-shot ood settings, demonstrating the robustness of our algorithm. ## conclusion in this paper, we have investigated few-shot ood intent detection under a more challenging setting. we have conducted a pilot study to identify the key challenge for this problem, which is in improving the in-distribution (id) expressiveness during training. to this end, we have proposed a promising approach to enrich the id training set by sampling from a denoising autoencoder trained with only a few examples. the enlarged training set enables to train a well-performing (k +1)-way classifier. our proposed approach has been validated by extensive experiments on real-world benchmarks.