Semi-supervised Intent Discovery with Contrastive Learning (2021)

Abstract: User intent discovery is a key step in developing a Natural Language Understanding (NLU) module at the core of any modern Conversational AI system. Typically, human experts review a representative sample of user input data to discover new intents, which is subjective, costly, and error-prone. In this work, we aim to assist the NLU developers by presenting a novel method for discovering new intents at scale given a corpus of utterances. Our method utilizes supervised contrastive learning to leverage information from a domain-relevant, already labeled dataset and identifies new intents in the corpus at hand using unsupervised K-means clustering. Our method outperforms the state-of-the-art by a large margin up to 2% and 13% on two benchmark datasets, measured by clustering accuracy. Furthermore, we apply our method on a large dataset from the travel domain to demonstrate its effectiveness on a real-world use case.

## introduction conversational ai systems such as chatbots and virtual assistants are widely used in a variety of applications to assist users. natural language understanding (nlu), as an integral part of a conversational ai system, is the process of classifying the user's input into a set of pre-defined categories, generally referred to as intents. in most applications, the task of nlu is achieved by developing a supervised text classification model. understanding and identifying user intents is key to developing intent classification models as it directly impacts the performance of the system. generally, the set of intents that can be recognized by the model is defined by human experts based on domain knowledge and business requirements. this process usually requires a significant amount of effort to manually review large-scale user input data. in addition, this task becomes increasingly complex as the number of potential intents in the dataset grows. to address these challenges, intent discovery methods that aim to detect new user intents, either automatically or semi-automatically, from a large number of unlabeled utterances, have been developed in recent years. a popular approach for intent discovery is based on unsupervised clustering algorithms. for instance, @xcite conduct hierarchical clustering on sentence representations from autoencoder based on a combination of word embeddings, pos tagging, keywords, and topic modeling. @xcite ) establish a multi-stage procedure which detects out-of-domain utterances with a classification model and then group the text with unknown intents by clustering. chatterjee and sengupta (2020) extend dbscan @xcite to discover intents in conversations based on sentence embeddings from universal sentence encoder @xcite . however, the unsupervised approaches often fail to produce highly accurate and granular intents. as such, researchers have developed semi-supervised methods to leverage existing domain-relevant labeled data in order to improve the intent discovery results. @xcite propose deep aligned clustering (dac) in a semi-supervised framework to leverage the prior information of known intents. dac uses bert to generate sentence representations. however, bert has poor performance with respect to generating semantically meaningful sentence representations @xcite . alternatively, @xcite use sentence-bert (sbert) to learn sentence representations but fail to achieve significantly better results using the dac algorithm without adjusting their training methods for sbert. new intents 1 3.encoding 4. k-means clustering 1.pairing mpnet mean pooling u contrastive loss sentence a mpnet mean pooling v sentence b sentence pairs sentence embedding label:1 label:0 2. training labeled utterances large amount of unlabeled utterance known intents figure 1: the architecture of the proposed method. first, we use labeled data to pair utterances based on their labels. then, we train the sentence transformer, mpnet, in the siamese network structure with contrastive loss to learn sentence representations using sentence pairs in a supervised way. next, we encode unlabeled data by the trained model. finally, we estimate the optimal values of the number of clusters and perform k-means to discover new intents from identified clusters. to address the limitations in the methods mentioned above and to further improve the results, we propose a novel semi-supervised method as illustrated in figure 1 . we use a siamese network @xcite to learn semantically meaningful sentence representations, in which a model is trained to differentiate pairs of sentences with the same intent and those with different intents. a pre-trained model that achieves the state-of-the-art performance among sentence transformers, mpnet @xcite , is used in the siamese network structure combined with contrastive learning to learn similar and dissimilar representations for sentences with the same and different intents respectively. k-means algorithm is then applied to cluster the unlabeled utterances based on their sentence representations for discovering new intents at scale. in addition, we propose a novel way to determine the optimal number of clusters k based on the concept of clustering stability, where k selection solely depends on the dataset at hand as opposed to relying on any prior knowledge. inspired by self-supervised learning with aligned pseudo labels introduced in dac @xcite , we further propose and experiment with four different pseudo label training (plt) strategies in the setting of siamese network, and examine the impact of plt in our experiments. the contributions of our work are summarized as follows. first, we propose a novel semi-supervised framework that learns sentence representations effectively via supervised contrastive learning and discovers new intents at scale via unsupervised clustering. second, we propose a simple and robust method based on clustering stability to determine the optimal number of clusters for k-means. third, we conduct extensive experiments to show that our method achieves state-of-the-art performance on two public benchmark datasets, widely used for intent discovery. in addition, we successfully apply our methods to a real-world dataset from an application in the travel domain. ## related work in this section we review the previous work focused on the main components of the intent discovery process in more detail. ## our approach in this section, we describe the proposed method in detail. as shown in figure 1 , we start by constructing the sentence pairs from our labeled dataset. then, we train a binary classifier with contrastive loss in a siamese network structure which uses a pre-trained mpnet model to identify if two utterances have the same intent or not. next, we encode a large number of unlabeled utterances using the trained mpnet. finally, we find the optimal k for k-means based on the concept of clustering stability and group the utterances using k-means to obtain new intents. ## conclusion and future work in this work, we propose a new semi-supervised framework to discover new intents by a sentence representation network via supervised contrastive learning followed by unsupervised k-means clustering. the method effectively leverages prior knowledge of existing intents to learn sentence representations and discovers new intents by grouping utterances with similar sentence representation. in the future, we will extend the work to discover intents with inherent hierarchy and automatically generate labels for new intents.