C o C o ID : Learning Contrastive Representations and Compact Clusters for Semi-Supervised Intent Discovery (2022)

Abstract: Intent discovery is to mine new intents from user utterances, which are not present in the set of manually predefined intents. Previous approaches to intent discovery usually automatically cluster novel intents with prior knowledge from intent-labeled data in a semi-supervised way. In this paper, we focus on the discriminative user utterance representation learning and the compactness of the learned intent clusters. We propose a novel semi-supervised intent discovery framework CoCoID with two essential components: contrastive user utterance representation learning and intra-cluster knowledge distillation. The former attempts to detect similar and dissimilar intents from a minibatch-wise perspective. The latter regularizes the predictive distribution of the model over samples in a cluster-wise way. We conduct experiments on both real-life challenging datasets (i.e., CLINC and BANKING) that are curated to emulate the true environment of commercial/production systems and traditional datasets (i.e., StackOverflow and DBPedia) to evaluate the proposed CoCoID. Experiment results demonstrate that our model substantially outperforms state-of-the-art intent discovery models (12 baselines) by over 1.4 ACC and ARI points and 1.1 NMI points across the four datasets. Further analyses suggest that CoCoID is able to learn contrastive representations and compact clusters for intent discovery.

## introduction intent discovery is to pinpoint novel intents from user utterances, which are not present in the set of predefined intents. discovering novel intents for goal-oriented dialogue has recently attracted growing attention and interest @xcite , not only because it is difficult to manually define all potential user intents for conversational agents deployed in a wide range of real-world scenarios, but also due to the cost of curating intent-labelled data. approaches to intent discovery usually include unsupervised clustering methods @xcite @xcite and semi-supervised methods @xcite @xcite . the former performs unsupervised clustering algorithms, e.g., k-means clustering, to group user utterances into clusters according to their underlying intents. the latter attempts to inject weakly supervised signals @xcite , or external knowledge @xcite into the procedure of clustering. compared with the unsupervised clustering methods, models that explore supervised signals or external knowledge are capable of discovering better user intents @xcite . in this paper, we follow the semi-supervised research philosophy for intent discovery. however, different from previous works @xcite , our focus lies on two aspects: user utterance representation and the compactness of clusters. for utterance representation, we have conducted a simple investigation in our preliminary experiments. with the deepaligned model @xcite , we have compared two different settings for user utterance representation learning: supervised simcse-bert_base foot_1 vs. simcse-bert_large foot_2 @xcite . the results are shown in to learn better user utterance representations for clustering and to improve the compactness of learned clusters, we propose cocoid that learns contrastive user utterance representations and compact clusters for semi-supervised intent discovery. to learn contrastive representations, we define a user utterance to be clustered as the anchor utterance and feed it to the feature extractor twice with different dropout masks. in doing so, we obtain two different representations for the anchor utterance and use them as positive pairs. other utterances in the same minibatch serve as negative samples. we then perform contrastive learning to improve utterance representations so that utterances with similar underlying intents tend to be close to each other while utterances with dissimilar intents are separated from each other. to improve the compactness of clusters, we propose an intra-cluster knowledge distillation (ickd) method. we randomly sample utterances from the cluster where the anchor utterance locates. the intent label of the anchor utterance is used to guide knowledge distillation from the anchor utterance to the sampled utterances. the motivation behind ickd is to help shorten the distance between utterances in the cluster, which can be also considered as a regularization strategy for the clustering model. in summary, our contributions are twofold. our work is related to intent discovery, contrastive learning and knowledge distillation. we briefly review these topics within the constraint of space. ## approach we elaborate the proposed cocoid in this section. the diagram of cocoid is shown in figure 1. it consists of three major components: semi-supervised clustering backbone, contrastive utterance representation learning and intra-cluster knowledge distillation. ## training objective contrastive utterance representation learning attempts to pull utterances with similar semantic intents together and push apart utterances with different intents in a minibatch-wise manner. by contrast, intra-cluster knowledge distillation is to shorten distances of utterances in the same cluster by regularizing predictive distributions in a cluster-wise way. these two approaches are complementary to each other and therefore can be combined in a unified framework cocoid. the final joint loss of cocoid with the two components can be formulated as: ## experiments we conducted a series of experiments on four widely-used datasets to evaluate the proposed co-coid. more details about the datasets and baselines can be found in the appendix a and b. ## analysis we carried out in-depth analyses to investigate how the proposed methods improve intent discovery. ## conclusions in this paper, we have presented a semi-supervised intent discovery framework cocoid. it consists of two essential components: contrastive user utterance representation learning and intra-cluster knowledge distillation. extensive experiments and analyses on real-life challenging datasets demonstrate that cocoid outperforms previous state-ofthe-art intent discovery models (12 baselines) and is able to learn contrastive representations and compact clusters.