A Probabilistic Framework for Discovering New Intents (2023)

Abstract: Discovering new intents is of great significance for establishing the Task-Oriented Dialogue System. Most existing methods either cannot transfer prior knowledge contained in known intents or fall into the dilemma of forgetting prior knowledge in the follow-up. Furthermore, these methods do not deeply explore the intrinsic structure of unlabeled data, and as a result, cannot seek out the characteristics that define an intent in general. In this paper, starting from the intuition that discovering intents could be beneficial for identifying known intents, we propose a probabilistic framework for discovering intents where intent assignments are treated as latent variables. We adopt the Expectation Maximization framework for optimization. Specifically, In the E-step, we conduct intent discovery and explore the intrinsic structure of unlabeled data by the posterior of intent assignments. In the M-step, we alleviate the forgetting of prior knowledge transferred from known intents by optimizing the discrimination of labeled data. Extensive experiments conducted on three challenging real-world datasets demonstrate the generality and effectiveness of the proposed framework and implementation.

## introduction unknown intent detection @xcite in the task-oriented dialogue system (tods) has gradually attracted more and more attention from researchers. however, detecting unknown intent is only the first step. for the tods, intent discovery is crucial but also more challenging. because the pre-defined intent set in the tods is limited to cover all intents, the tods should discover potential new intents automatically during interactions with the users. and as a practical matter, a large number of valuable unlabeled data will be generated within the interaction between users and the dialogue system. considering the limited labeled corpus and time-consuming annotating, which also requires expertise, the tods should adaptively discover intents from those unlabeled data with the aid of limited labeled data. just as discovering new intents plays a crucial role in establishing the tods, discovering new intents has raised a lot of research interest just like unknown intent detection. unsupervised cluster learning is a popular paradigm to solve this problem. specifically, previous works @xcite @xcite formulate intent discovery as an unsupervised clustering process. however, these methods mainly focus on constructing pseudo-supervised signals to guide the clustering process while neglecting the prior knowledge embedded in the available labeled data. in real user-facing scenarios, we often possess a small amount of labeled data in advance, which contains prior knowledge that can guide the intent discovery process, and a substantial volume of unlabeled data generated in the interaction with the dialogue system mentioned above, which contains both known intents and unknown intents to be discovered. how do discover intents in the unlabeled corpus using the labeled data? recently, the semisupervised methods @xcite have become popular. deepaligned @xcite is the most typical and has also inspired a series of effective works @xcite recently. deepaligned first generalizes the prior knowledge into the semantic features of unlabeled data by pre-training. then, to learn cluster-friendly representations, deepaligned assigns a pseudo label to each unlabeled utterance and re-trains the model under the supervision of those pseudo labels. @xcite . during clustering in deepaligned, we test the performance of the model on the verification set used in the transferring prior knowledge stage and show that with the advancement of clustering, the model constantly forgets the knowledge learned from labeled data. the brown line represents the baseline obtained by the model after transferring prior knowledge. in contrast, our method (red) can alleviate forgetting well. see section 5.6 for more discussion. nevertheless, deepaligned suffers from many problems. firstly, when the model is re-trained with the pseudo supervision signal, the model will forget the knowledge transferred in the transferring stage, which is demonstrated in figure 1 . then, the model could be misled by inaccurate pseudo labels, particularly in large-sized intent space @xcite . more importantly, softmax loss formed by pseudo labels cannot explore the intrinsic structure of unlabeled data, so it can not provide accurate clustering supervised signals. different from the previous methods, we start from the intuition that the intent discovery should not damage the identification of the known intents. ideally, the two processes should achieve a win-win situation. the knowledge contained in labeled data corpus (as known intents) can be used to guide the discovery, and the information learned from the unlabeled corpus during discovery could improve the identification of the known intents. therefore, with the help of optimizing the identification of labeled data given the whole data corpus, we propose a principled probabilistic framework for intent discovery, where intent assignments as a latent variable. we adopt expectation maximization as a principal template for optimizing this typical latent variable model. specifically, in the e-step, we use the current model to discover intents and calculate a specified posterior probability of intent assignments to explore the intrinsic structure of data. in the m-step, the probability of identification of labeled data including those newly discovered from unlabeled data, and the posterior probability of intent assignments, which is to help learn friendly-discovery features, are maximized simultaneously to optimize and update model parameters. extensive experiments conducted in three benchmark datasets demonstrate our method can achieve substantial improvements over strong baselines. our contributions are as follows: ## related work our work is mainly related to two lines of research: unsupervised and semi-supervised clustering. unsupervised clustering extracting meaningful information from unlabeled data has been studied for a long time. traditional approaches like k-means @xcite and agglomerative clustering (ac) @xcite are seminal but hardly perform well in highdimensional space. recent efforts are devoted to using the deep neural network to obtain good clustering representations. @xcite propose deep embedded cluster (dec) to learn and refine the features iteratively by optimizing a clustering objective based on an auxiliary distribution. unlike dec, @xcite propose deep clustering network (dcn) that performs nonlinear dimensionality reduction and k-means clustering jointly to learn friendly representation. @xcite (dac) apply unsupervised clustering to image clustering and proposes a binary-classification framework that uses adaptive learning for optimization. then, deepcluster @xcite proposes an end-to-end training method that performs cluster assignments and representation learning alternately. however, the key drawback of unsupervised methods is their incapability of taking advantage of prior knowledge to guide the clustering. semi-supervised clustering by virtue of a few labeled data, semi-supervised clustering usually produces better results compared with unsupervised counterparts. pck-means @xcite proposes that the clustering can be supervised by pairwise constraints between samples in the dataset. kcl @xcite transfers knowledge in the form of pairwise similarity predictions first and learns a clustering network to transfer learning. along this line, mcl @xcite further formulates multi-classification as meta classification that predicts pairwise similarity and generalizes the paradigm to various settings. dtc @xcite extends the dec algorithm and proposes a mechanism to estimate the number of new images categories using labeled data. when it comes to the field of text clustering, cdac+ (lin et al., 2020) combines the pairwise constraints and target distribution to discover new intents while deepaligned @xcite introduces an alignment strategy to improve the clustering consistency. recently, scl @xcite incorporates a strong backbone mpnet in the siamese network structure with pairwise contrastive loss to learn the sentence representations. similarly, mtp @xcite enhances sentence representation through multi-task pre-training strategy and extra data. although these methods take known intents into account, they may suffer from knowledge forgetting during the training process. more importantly, these methods are insufficient in the probe into the intrinsic structure of unlabeled data, making it hard to distinguish the characteristics that form an intent. ## conclusion in this paper, we provide a probabilistic framework for intent discovery. this is the first complete theoretical framework for intent discovery. we also provide an efficient implementation based on this proposed framework. compared with the existing methods, our method effectively alleviates the forgetting of prior knowledge transferred from known intents and provides intensive clustering supervised signals for discovering intents. extensive experiments conducted in three challenging datasets demonstrate our method can achieve substantial improvements. the subsequent analysis also shows that our method can better estimate the number of intents and adapt to various conditions. in the future, we will try different methods to perform intent assignments and explore more methods to approximate p(y l |z, d; θ) and p(z|d; θ).