Modeling Discriminative Representations for Out-of-Domain Detection with Supervised Contrastive Learning (2021)

Abstract: Detecting Out-of-Domain (OOD) or unknown intents from user queries is essential in a task-oriented dialog system. A key challenge of OOD detection is to learn discriminative semantic features. Traditional cross-entropy loss only focuses on whether a sample is correctly classified, and does not explicitly distinguish the margins between categories. In this paper, we propose a supervised contrastive learning objective to minimize intra-class variance by pulling together in-domain intents belonging to the same class and maximize inter-class variance by pushing apart samples from different classes. Besides, we employ an adversarial augmentation mechanism to obtain pseudo diverse views of a sample in the latent space. Experiments on two public datasets prove the effectiveness of our method capturing discriminative representations for OOD detection.

## introduction detecting out-of-domain (ood) or unknown intents from user queries is an essential component in a task-oriented dialog system @xcite @xcite . it aims to know when a user query falls outside their range of predefined supported intents to avoid performing wrong operations. different from normal intent detection tasks, we do not know the exact number of unknown intents in practical scenarios and can barely annotate extensive ood samples. lack of real ood examples leads to poor prior knowledge about these unknown intents, making it challenging to identify ood samples in the task-oriented dialog system. previous methods of ood detection can be generally classified into two types: supervised and unsupervised ood detection. supervised ood detection @xcite @xcite @xcite represents that there are extensive labeled ood samples in the training data. in contrast, unsupervised ood detection @xcite @xcite @xcite @xcite means no labeled ood samples except for labeled in-domain data. specifically, for supervised ood detection, @xcite ; @xcite , form a (n +1)-class classification problem where the (n + 1)-th class represents the unseen intents. further, @xcite uses labeled ood data to generate an entropy regularization term to enforce the predicted distribution of ood inputs closer to the uniform distribution. however, these methods heavily rely on large-scale time-consuming labeled ood data. compared to these supervised methods, unsupervised ood detection first learns discriminative intent representations via in-domain (ind) data, then employs detecting algorithms, such as maximum softmax probability (msp) @xcite , local outlier factor (lof) @xcite , gaussian discriminant analysis (gda) @xcite to compute the similarity of features between ood samples and ind samples. in this paper, we focus on the unsupervised ood detection. a key challenge of unsupervised ood detection is to learn discriminative semantic features via ind data. we hope to cluster the same type of ind intents more tightly and separate different types of ind intents further. traditional softmax loss @xcite only focuses on whether the sample is correctly classified, and does not explicitly distinguish the relationship between categories. further, @xcite proposes a large margin cosine loss (lmcl) @xcite which maximizes the decision margin in the latent space. lmcl forces the model to not only classify correctly but also maximize inter-class variance and minimize intra-class variance. following the similar motivation, we aim to pull intents belonging to the same class together while simultaneously pushing apart samples from different classes to further model discriminative semantic features. in this paper, we propose a supervised contrastive learning (scl) model to learn discriminative semantic intent representation for ood detection. scl aims to minimize intra-class variance by pulling together ind intents belonging to the same class and maximize inter-class variance by pushing apart samples from different classes. empirical results demonstrate the effectiveness of discriminative representation for ood detection. besides, to enhance the diversity of data augmentation in scl, we employ an adversarial attack mechanism to obtain pseudo hard positive samples in the latent space by computing model-agnostic adversarial worst-case perturbations to the inputs. our contributions are three-fold: (1) to the best of our knowledge, we are the first to apply supervised contrastive learning to ood detection. (2) compared to cross-entropy (ce) loss, scl+ce can maximize inter-class variance and minimize intra-class variance to learn discriminative semantic representation. (3) extensive experiments and analysis on two public datasets demonstrate the effectiveness of our method. ## methodology overall architecture fig 1 shows the overall architecture of our proposed method. as fig 1(a ) displays, we first train an ind intent classifier us-ing ce or scl+ce objectives in the training stage. then in the test stage, we extract the intent feature of a test sample and employ the detection algorithms msp @xcite , lof @xcite or gda @xcite to detect ood. 2 fig 1(b) demonstrates the effectiveness of our method capturing discriminative intent representations, where scl+ce can maximize interclass variance and minimize intra-class variance. supervised contrastive learning we first review the classic cross-entropy (ce) loss and its improved version, large margin cosine loss (lmcl). then we explain our supervised contrastive loss (scl) in detail. given an ind sample x i and its intent label y i , we adopt a bilstm @xcite or bert @xcite encoder to get the intent representation s i . the ce loss and lmcl are defined as follows foot_1 : as fig 1(b) shows, scl aims to pull together ind intents belonging to the same class and pushing apart samples from different classes, which helps recognize ood intents near the decision boundary. in the implementation, we first pre-train the intent classifier using scl, then finetune the model using ce or lmcl, both on the ind data. we compare iterative training and joint training in the appendix. ## conclusion in this paper, we focus on the unsupervised ood detection where no labeled ood data exist. to learn discriminative semantic intent representations via in-domain data, we propose a novel supervised contrastive learning loss (scl). scl aims to minimize intra-class variance by pulling together indomain intents belonging to the same class and maximize inter-class variance by pushing apart samples from different classes. experiments and analysis confirm the effectiveness of scl for ood detection. we hope to provide new guidance for future ood detection work.