A Deep Generative Distance-Based Classifier for Out-of-Domain Detection with Mahalanobis Space (2020)

Abstract: Detecting out-of-domain (OOD) input intents is critical in the task-oriented dialog system. Different from most existing methods that rely heavily on manually labeled OOD samples, we focus on the unsupervised OOD detection scenario where there are no labeled OOD samples except for labeled in-domain data. In this paper, we propose a simple but strong generative distance-based classifier to detect OOD samples. We estimate the class-conditional distribution on feature spaces of DNNs via Gaussian discriminant analysis (GDA) to avoid over-confidence problems. And we use two distance functions, Euclidean and Mahalanobis distances, to measure the confidence score of whether a test sample belongs to OOD. Experiments on four benchmark datasets show that our method can consistently outperform the baselines.

## introduction task-oriented dialog systems such as google's dialogflow or amazon's lex have become omnipresent to let people interact with machines using natural language. detecting unknown or ood (out-of-domain) intents from user queries is an essential component that aims to know when a query falls outside their range of predefined supported intents. correctly identifying out-of-scope cases is thus crucial in deployed systems-both to avoid performing the wrong action and also to identify potential future directions for development. different from traditional text classification tasks, the exact number of unknown intents in practical scenarios is hard to estimate by domain experts and lack of real ood examples always leads to poor prior knowledge about these unknown intents. these characteristics make it challenging to identify ood samples in the task-oriented dialog system. we classify the existing methods of detecting ood intents into two main categories, supervised and unsupervised ood detection. supervised ood detection @xcite @xcite @xcite represents that there are extensive labeled ood samples in the training data while unsupervised ood detection @xcite @xcite @xcite @xcite means few or no labeled ood samples except for labeled in-domain data. unsupervised ood detection makes it more complicated to identify unknown intents due to unseen and diverse semantic expressions. besides, in the practical scenario, collecting large-scale ood data is usually difficult and expensive compared to in-domain predefined intent data, especially when dealing with the rapidly evolving open-world environment. in this paper, we focus on the latter ood setting, unsupervised ood detection. for supervised ood detection, classical methods such as @xcite , form a (m + 1)-class classification problem where the (m + 1)-th class represents the unseen intents. further, @xcite uses labeled ood data to generate an entropy regularization term to enforce the predicted distribution of ood inputs closer to the uniform distribution. the critical drawback is that collecting ood data is usually labor-intensive and they do not guarantee similar semantics. typically, embedding i want to listen to seventies music playmusic book a spot for 3 in mt bookrestaurant rate this series a 5 ratebook add this track to my global funk ood samples train test bilstm encoder softmax / lcml gda + distant function a) the overall architecture b) generative distance-based ood detector playmusic bookrestaurant ratebook ood detection boundary confidence score ood samples euclidean distance based mahalanobis distance based in-domain classification out-of-domain detection i rate secret water as a 4 id samples (ratebook) figure 1 : the architecture of our proposed method. we first extract intent representation by a intent classifier pre-trained on in-domain data. then, we use distance functions in latent space to detect unknown intents for test. in this paper, we propose a simple generative distance-based classifier to detect ood samples. our method can be applied to any pre-trained softmax neural classifier without re-training and avoid the issue of overconfident predictions. specifically, we first estimate the class-conditional distribution on feature spaces of dnns via gaussian discriminant analysis (gda). then we use two distance functions, euclidean and mahalanobis distances, to measure the confidence score of whether a test sample belongs to ood. our contributions are three-fold: (1) we propose a generative distance-based classifier for ood detection. (2) apart from traditional euclidean distance, we introduce mahalanobis distance under gda to takes into account the correlations between features and get better results. (3) experiments conducted on four benchmark ood datasets show the effectiveness of the proposed method. ## approach fig 1(a) shows the overall architecture of our proposed method. we first train an intent classifier on in-domain data and then use a generative distance-based ood detector to identify unknown intents for test. fig 1( @xcite shows the difference between euclidean and mahalanobis distances in latent space. ## conclusion in this paper, we focus on the unsupervised ood detection and propose a simple but strong generative distance-based classifier to detect ood samples. we combine the advantages of dnns and gaussian discriminant analysis (gda) to avoid previous over-confidence problems and apply mahalanobis distance to measure the confidence score. experiments show that our method achieves better performance and is more robust to data imbalance and poor feature extractor as well as the few-shot scenario.