Watch the Neighbors: A Unified K-Nearest Neighbor Contrastive Learning Framework for OOD Intent Discovery (2022)

Abstract: Discovering out-of-domain (OOD) intent is important for developing new skills in task-oriented dialogue systems. The key challenges lie in how to transfer prior in-domain (IND) knowledge to OOD clustering, as well as jointly learn OOD representations and cluster assignments. Previous methods suffer from in-domain overfitting problem, and there is a natural gap between representation learning and clustering objectives. In this paper, we propose a unified K-nearest neighbor contrastive learning framework to discover OOD intents. Specifically, for IND pre-training stage, we propose a KCL objective to learn inter-class discriminative features, while maintaining intra-class diversity, which alleviates the in-domain overfitting problem. For OOD clustering stage, we propose a KCC method to form compact clusters by mining true hard negative samples, which bridges the gap between clustering and representation learning. Extensive experiments on three benchmark datasets show that our method achieves substantial improvements over the state-of-the-art methods.

## introduction out-of-domain (ood) intent discovery aims to group new unknown intents into different clusters, which helps identify potential development directions and develop new skills in a task-oriented dialogue system @xcite @xcite . different from traditional text clustering task, ood discovery considers how to leverage the prior knowledge of known in-domain (ind) intents to enhance discovering unknown ood intents, which makes it challenging to directly apply existing clustering algorithms @xcite @xcite to the ood discovery task. the key challenges of ood intent discovery come from two aspects: (1) knowledge transferability. it requires transferring in-domain prior knowledge to help downstream ood clustering. early unsupervised intent discovery methods @xcite @xcite only model unlabeled ood data but ignore prior knowledge of labeled in-domain data thus suffer from poor performance. then, recent work @xcite @xcite focus more on the semi-supervised setting where they firstly pre-train an in-domain intent classifier then perform clustering algorithms on extracted ood intent representations by the pre-trained ind intent classifier. for example, @xcite ; @xcite pretrain a bert-based @xcite in-domain intent classifier using cross-entropy (ce) classification loss. @xcite further proposes a supervised contrastive learning (scl) @xcite objective to learn discriminative intent representations. (2) jointly learning representations and cluster assignments. it's important to learn ood intent features while performing clustering. @xcite uses ood representations to calculate the similarity of ood sample pairs as weak supervised signals. the gap between pretrained ind features and unseen ood data makes it hard to generate high-quality pairwise pseudo labels. then, @xcite proposes an iterative clustering method, deepaligned, to obtain pseudo cluster labels by k-means @xcite . it performs representation learning and cluster assignment in a pipeline way. further, @xcite introduces a multi-head contrastive clustering framework to jointly learn representations and cluster assignments using contrastive learning. however, all of these methods still suffer from two problems: (1) in-domain overfitting: the state-of-the-art (sota) ood intent discovery methods @xcite adopt general supervised pre-training objectives such as ce and scl for ind pre-training, but none of them consider the following question: what kind of intent representation is more generalized to transfer to downstream ood clustering? although ce and scl are effective for classifying known ind classes, such learned representations are poor for downstream transfer. @xcite find larger intra-class diversity helps transfer. ce and scl tend to pull all samples from the same class together to form a narrow intra-class distribution, thus ignore the intra-class diverse features, which makes the learned representations unfavorable to transfer to the downstream ood clustering. (2) gap between clustering objectives and representation learning. learning ood intent representations is key for achieving efficient clustering. @xcite can't align learning intent features with clustering because the two processes are iterative in a pipeline way. @xcite further proposes dkt to jointly learn representation and cluster assignment using contrastive learning @xcite . however, such a contrastive objective of learning intent features pushes apart representations of different samples in the same class, which impair the clustering goal where samples within the same class should be compact. to solve the two problems, we propose a unified k-nearest neighbor contrastive learning framework for ood discovery (kcod). for the indomain overfitting issue, we propose a simple knearest neighbor contrastive learning objective (kcl) for ind pre-training in fig 1 @xcite . compared with scl, we only take the k nearest samples of the same class as positive samples, which helps to increase the intra-class variance while maintaining a large inter-class variance. larger intra-class diversity helps downstream transfer. for the gap between clustering and representation learning objectives, we propose a k-nearest neighbor contrastive clustering method (kcc) for ood clustering in fig 1 @xcite . traditional instance-wise contrastive learning only regards an anchor and its augmented sample as positive pairs and pushes apart representations of different samples even in the same class. in contrast, kcc firstly filters out false negative samples that belong to the same class as the anchor and then selects the k nearest neighbors as true negatives. we aim to mine high-confident hard negatives and learn compact intent representations for ood clustering to form clear cluster boundaries. our contributions are three-fold: (1) we propose a unified k-nearest neighbor contrastive learning (kcod) framework for ood discovery, which aims to achieve better knowledge transfer and learn better clustering representations. (2) we propose a k-nearest neighbor contrastive learning (kcl) objective for ind pre-training, and a k-nearest neighbor contrastive clustering (kcc) method for ood clustering, which solve "in-domain overfitting" problem and bridge the gap between clustering objectives and representation learning. (3) experiments and analysis demonstrate the effectiveness of our method for ood discovery. ## approach ood discovery assumes there is a set of labeled indomain data and unlabeled ood data foot_0 . our goal is to cluster ood concepts from unlabeled ood data using prior knowledge from labeled ind data. the overall architecture of our kcod is shown in fig 2 , including knn contrastive ind pre-training and knn contrastive ood clustering. ind pretraining firstly gets generalized intent representa-bert encoder pooling layer in-domain samples out-of-domain samples softmax classifier cluster-level head instance-level head cross-entropy loss cluster-level contrastive loss kcc loss kcl loss kcl kcc i z i f i g threshold pull together push away samples for different intents stage 1: ind pre-training stage 2: ood clustering tions via our proposed kcl objective and then ood clustering uses kcc to group ood intents into different clusters. ## related work true clusters predict clusters tering. recent work @xcite @xcite performs semi-supervised clustering using labeled in-domain data. to transfer intent representation, lin et al. (2020); @xcite pre-train a bert encoder using crossentropy loss, then @xcite uses scl @xcite to learn discriminative features. all the models face the challenge of indomain overfitting issues where representations learned from ind data will degrade for ood data. thus, we propose a kcl loss to keep large interclass variance and help downstream transfer. for ood clustering, @xcite use kmeans to learn cluster assignments but ignore joint learning intent representations. @xcite uses contrastive clustering where the instance-level contrastive loss for learning intent features has a gap with the cluster-level loss for clustering. therefore, we propose the kcc method to mine hard negatives to form clear cluster boundaries. contrastive learning contrastive learning (cl) is widely used in self-supervised learning @xcite @xcite . zeng et al. (2021a); liu et al. (2021b); zeng et al. (2021b) apply it to ood detection. @xcite proposes a knn-contrastive learning method for ood detection. it aims to learn discriminative semantic features that are more conducive to anomaly detection. in contrast, our method uses a unified k-nearest neighbor contrastive learning framework for ood discovery where kcl increases intra-class diversity and helps downstream transfer, and kcc learns compact intent representations for ood clustering to form clear cluster boundaries. @xcite uses contrastive clustering for ood discovery. but original instance-level cl pushes apart different instances of the same intent which is against clustering. thus, we use a simple k-nearest sampling mechanism to separate clusters and form clear boundaries. ## conclusion in this paper, we propose a unified k-nearest neighbor contrastive learning (kcod) framework for ood intent discovery. we design a kcl objective for ind pre-training, and a kcc method for ood clustering. experiments on three benchmark datasets prove the effectiveness of our method. and extensive analyses demonstrate that kcl is helpful for learning intra-class diversity knowledge and alleviating the problem of intra-domain overfitting, and kcc is beneficial for forming compact clusters, effectively bridging the gap between clustering and representation learning. we hope to explore more self-supervised learning methods for ood discovery in the future.