Intent Detection and Discovery from User Logs via Deep Semi-Supervised Contrastive Clustering (2022)

Abstract: Intent Detection is a crucial component of Dialogue Systems wherein the objective is to classify a user utterance into one of multiple pre-defined intents. A pre-requisite for developing an effective intent identifier is a training dataset labeled with all possible user intents. However, even skilled domain experts are often unable to foresee all possible user intents at design time and for practical applications, novel intents may have to be inferred incrementally on-the-fly from user utterances. Therefore, for any real-world dialogue system, the number of intents increases over time and new intents have to be discovered by analyzing the utterances outside the existing set of intents. In this paper, our objective is to i) detect known intent utterances from a large number of unlabeled utterance samples given a few labeled samples and ii) discover new unknown intents from the remaining unlabeled samples. Existing SOTA approaches address this problem via alternate representation learning and clustering wherein pseudo labels are used for updating the representations and clustering is used for generating the pseudo labels. Unlike existing approaches that rely on epoch wise cluster alignment, we propose an end-to-end deep contrastive clustering algorithm that jointly updates model parameters and cluster centers via supervised and self-supervised learning and optimally utilizes both labeled and unlabeled data. Our proposed approach outperforms competitive baselines on five public datasets for both settings: (i) where the number of undiscovered intents are known in advance, and (ii) where the number of intents are estimated by an algorithm. We also propose a human-in-the-loop variant of our approach for practical deployment which does not require an estimate of new intents and outperforms the end-to-end approach.

## introduction modern dialogue systems @xcite are increasingly reliant on intent detection to classify a user utterance into one of the multiple known user intents. intent detection is typically modeled as a multi-class classification problem where labeled data comprising of utterances for each known intent is manually created by domain experts. however, most real-world applications have to cope with evolving user needs and new functionality is routinely introduced into the dialogue system resulting in a continuously increasing number of intents over time. even for seasoned domain experts estimating future user requirements at design time is challenging and these often have to be discovered from recent user logs which contain information corresponding to past user utterances, model response (i.e., predicted intent), implicit (confidence or softmax probability), and explicit (user clicks on a thumbs up or thumbs down icon) feedback as shown in fig 1 . the intent detection model presented in fig. 1 is trained on two initial intents (i 1 , i 2 ) from the banking domain using labeled data created by domain experts. filtered user logs containing implicit and explicit feedback were shared with domain experts, who, discovered two new intents (i 3 , i 4 ) and mapped the filtered utterances to these new intents. additionally, experts also have to identify and discard utterances that are outside the domain of the dialog system. the remaining user logs are the primary source of evolving user needs and the process of identifying utterances belonging to "new intents"/"unknown intents" from user logs is referred to as intent discovery/intent mining @xcite . however, manually monitoring user logs is not scalable and our objective in this paper is to present novel sota techniques to perform automated and semi-automated intent detection and discovery over user logs given only a few labeled utterances from known intents in addition to unlabeled utterances from both known and unknown intents. several classical @xcite @xcite and deep learning @xcite based clustering methods have been used for intent discovery. chatterjee and sengupta (2020) modeled intent discovery from unlabeled utterances as an unsupervised clustering problem and proposed a variant of dbscan @xcite for clustering but do not employ any representation learning and rely heavily on manual evaluation. @xcite use a contrastive learning @xcite based unsupervised approach for joint representation learning and clustering where performance largely depends on the quality of an auxiliary target distribution @xcite . @xcite and @xcite model intent detection and discovery as a semi-supervised learning problem where the objective is to detect known intents and discover new intents given 1) a few labeled utterances from known intents along with 2) unlabeled utterances from known and new intents. this is similar to our problem of intent detection and discovery from user logs. deep-aligned @xcite is the current sota approach for intent detection and discovery alternately performing representation learning and clustering by utilizing pseudo-labeled data obtained from clustering for representation learning. deep-aligned uses k-means @xcite as the clustering algorithm of choice and updates a bert @xcite backbone's parameters in the process. as k-means may assign different cluster ids to the same set of data points over different iterations the authors propose an alignment algorithm to align clusters obtained in consecutive epochs. thus, an incorrect cluster alignment over epochs may lead to a significant drop in clustering accuracy. additionally, they make the unrealistic assumption of a uniform distribution over intents to estimate the number of intents. in this paper, we propose a novel end-toend deep semi-supervised contrastive clustering (dsscc-e2e) algorithm for intent detection and discovery from user logs. dsscc-e2e is motivated by recent advances in self-supervised @xcite @xcite and supervised contrastive learning @xcite applied to computer vision and natural language processing. we model intent detection and discovery as a form of semi-supervised contrastive clustering wherein we jointly update backbone representations and cluster centers by minimizing the distance between the distribution over clusters of similar utterances and maximizing the same for dissimilar utterances via a semisupervised variant of supervised contrastive (sup-con) loss @xcite . for contrastive learning, we use the contextual augmenter @xcite to create pairs of augmentations (positive pairs/ similar utterances) corresponding to each unlabeled utterance from known and unknown intents. for labeled utterances, we use pairs of utterances with the same intent to create positive pairs. to improve the accuracy of intent detection, we update representations based on labeled utterances by minimizing the cross-entropy loss. to avoid the trivial solution that assigns all utterances to the same cluster @xcite , similar to van gansbeke et al. (2020); @xcite we also add an entropy term to the loss function which distributes utterances uniformly across the clusters. existing semi-supervised approaches @xcite including dsscc-e2e and some unsupervised approaches @xcite for intent discovery require an estimate of the number of new intents (m) present in the user logs. incorrect estimates for m can lead to noisy clusters (i.e., a cluster which contains utterances from multiple intents), which then require substantial manual effort to split cleanly. unsupervised approaches @xcite often lead to a large number of clusters due to poor semantic utterance representations. for practical deployment, we propose a human-in-loop variant of dsscc-e2e called dsscc-hil which does not estimate m and instead creates multiple dense clusters via dbscan. domain experts can then merge these clusters with minimal manual effort such that each merged cluster represents an intent. our key contributions are: (1) we propose a novel deep semi-supervised contrastive clusteringbased approach for intent detection and discovery from user logs. (2) dsscc-e2e does not require epoch-wise cluster alignment, is end-to-end trainable, and fully utilizes both the labeled and unlabeled utterances for novel intent discovery. 2 related work ## proposed approach as shown in fig. 2 , we propose a two-phase algorithm for intent detection and discovery from user logs. in the first phase of dsscc, the parameters of a pre-trained language model (plm) are updated based on labeled data from known intents. in the second phase, we perform joint representation learning and clustering by updating the cluster centers and plm parameters via semi-supervised contrastive learning. in addition to contrastive learning, dsscc also uses labeled utterances to update plm representations via cross-entropy loss. further, the entropy of the intent distribution is maximized to . alleviate the issue of empty clusters, i.e., when all utterances become part of a single cluster. in a realistic scenario, the number of unknown intents m present in user logs is not known in advance so we use the approach proposed by @xcite to estimate m and use dsscc-e2e for intent detection and discovery. the approach uses a plm to get utterance representations and run k-means with a very high value of @xmath0. however in practice, we find that even this estimation method is often incorrect, and in section 4.5, we describe how we derive dsscc-hil from dsscc-e2e to address this problem with a small amount of manual supervision. we use the hungarian @xcite algorithm to align clusters with known and unknown intents and report performance. in the rest of the paper, we use dsscc and dsscc-e2e interchangeably and use dsscc-hil to refer to the human-in-the-loop variant of our approach. ## experimental setup in this section, we describe the various baselines, datasets, and evaluation metrics used in our experiments. ## results and discussion we present results comparing dsscc with unsupervised and semi-supervised clustering approaches for scenarios where 1) the system is aware of the number of unknown intents, 2) the system is unaware of the number of unknown intents. we further report results of dsscc-hil and compare it with dsscc-e2e. ## conclusion in this work, we propose a semi-supervised contrastive learning approach for intent detection and discovery from user logs. the proposed approach optimally utilizes both labeled and unlabeled utterances to outperform sota approaches for both scenarios where the total number of intents is either known in advance or has to be estimated. we also propose a variant of our approach which does not need to estimate the number of new intents and yields pure clusters which are merged by domain experts based on the cluster descriptions. future work will focus on, (i) "how to get better cluster descriptions?" (ii) "how to optimally select query and corresponding candidate clusters?" and (iii) "how to discover new intents with minimum human effort with a long-tail distribution over new intents in user logs?"