Out-of-Scope Intent Detection with Self-Supervision and Discriminative Training (2021)

Abstract: Out-of-distribution (OOD) intent detection is of practical importance in task-oriented dialogue systems. Since the distribution of outlier utterances is arbitrary and unknown in the training stage, existing methods commonly rely on strong assumptions on data distribution such as mixture of Gaussians to make inference, resulting in either complex multi-step training procedures or hand-crafted rules such as confidence threshold selection for outlier detection.
In this paper, we propose a simple yet effective method to train an OOD intent classifier in a fully end-to-end manner by simulating the test scenario in training, which requires no assumption on data distribution and no additional post-processing or threshold setting. Specifically, we construct a set of pseudo outliers in the training stage, by generating synthetic outliers using inliner features via self-supervision and sampling OOD sentences from easily available open-domain datasets. The pseudo outliers are used to train a discriminative classifier that can be directly applied to and generalize well on the test task. We evaluate our method extensively on four benchmark dialogue datasets and observe significant improvements over state-of-the-art approaches.
The source code has been made available at \url{https://github.com/liam0949/DCLOOS}.

## introduction conversational system is becoming an indispensable component in a variety of ai applications and acts as an interactive interface provided to users to improve user experience. language understanding is essential for conversational systems to provide appropriate responses to users, and intent detection is usually the first step of language understanding. the primary goal is to identify diverse intentions behind user utterances, which is often formalized as a classification task. however, intent classes defined during training are inevitably inadequate to cover all possible user intents at the test stage due to the diversity and randomness of user utterances. hence, out-of-distribution (ood) intent detection is essential, which aims to develop a model that can accurately identify known (seen in training) intent classes while detecting the ood classes that are not encountered during training. due to the practical importance of ood intent detection, recent efforts have attempted to solve this problem by developing effective intent classification models. in general, previous works approach this problem by learning decision boundaries for known intents and then using some confidence measure to distinguish known and unknown intents. for examples, lmcl @xcite learns the decision boundaries with a margin-based optimization objective, and seg @xcite assumes the known intent classes follow the distribution of mixture of gaussians. after learning the decision boundaries, an off-the-shell outlier detection algorithm such as lof @xcite is commonly employed to derive confidence scores @xcite @xcite . if the confidence score of a test sample is lower than a predefined threshold, it is identified as an outlier. however, it may be problematic to learn decision boundaries solely based on the training examples of known intent classes. first, if there are sufficient training examples, the learned decision boundaries can be expected to generalize well on known intent classes, but not on the unknown. therefore, extra steps are required in previous methods, such as using an additional outlier detection algorithm at the test stage or adjusting the confidence threshold by cross-validation. on the other hand, if there are not sufficient training examples, the learned boundaries may not generalize well on both known and unknown intents. as a result, these methods often underperform when not enough training data is given. hence, it is important to provide learning signals of unknown intents at the training stage to overcome these limitations. in contrast to previous works, we adopt a different approach by explicitly modeling the distribution of unknown intents. particularly, we construct a set of pseudo ood examples to aid the training process. we hypothesize that in the semantic feature space, real-world outliers can be well represented in two types: "hard" outliers that are geometrically close to the inliers and "easy" outliers that are distant from the inliners. for the "hard" ones, we construct them in a self-supervised manner by forming convex combination of the features of inliers from different classes. for the "easy" ones, the assumption is that they are very unrelated to the known intent classes, so they can be used to simulate the randomness and diversity of user utterances. they can be easily constructed using public datasets. for example, in our experiments, we randomly collect sentences from datasets of other nlp tasks such as question answering and sentiment analysis as open-domain outliers. in effect, by constructing pseudo outliers for the unknown class during training, we form a consistent (k + 1) classification task (k known classes + 1 unknown class) for both training and test. our model can be trained with a cross-entropy loss and directly applied to test data for intent classification and outlier detection without requiring any further steps. as shown in figure 1 (better view in color and enlarged), our method can learn better utterance representations, which make each known intent class more compact and push the outliers away from the inliers. our main contributions are summarized as follows. ## experiments in this section, we present the experimental results of our proposed method on the targeted task of ood intent detection. given a test set comprised of known and ood intent classes, the primary goal of an ood intent detection model is to assign correct intent labels to utterances in the test set. notice that the unknown intent label c ood is also included as a special class for prediction. ## conclusion we have proposed a simple, effective, and efficient approach for out-of-distribution intent detection by overcoming the limitation of previous methods via matching train-test conditions. particularly, at the training stage, we construct self-supervised and open-domain outliers to improve model generalization and simulate real outliers in the test stage. extensive experiments on four dialogue datasets show that our approach significantly outperforms state-of-the-art methods. in the future, we plan to investigate the theoretical underpinnings of our approach and apply it to more applications.