New Intent Discovery with Pre-training and Contrastive Learning (2022)

Abstract: New intent discovery aims to uncover novel intent categories from user utterances to expand the set of supported intent classes. It is a critical task for the development and service expansion of a practical dialogue system. Despite its importance, this problem remains under-explored in the literature. Existing approaches typically rely on a large amount of labeled utterances and employ pseudo-labeling methods for representation learning and clustering, which are label-intensive, inefficient, and inaccurate. In this paper, we provide new solutions to two important research questions for new intent discovery: (1) how to learn semantic utterance representations and (2) how to better cluster utterances. Particularly, we first propose a multi-task pre-training strategy to leverage rich unlabeled data along with external labeled data for representation learning. Then, we design a new contrastive loss to exploit self-supervisory signals in unlabeled data for clustering. Extensive experiments on three intent recognition benchmarks demonstrate the high effectiveness of our proposed method, which outperforms state-of-the-art methods by a large margin in both unsupervised and semi-supervised scenarios. The source code will be available at https://github.com/zhang-yu-wei/MTP-CLNN .

## introduction why study new intent discovery (nid)? recent years have witnessed the rapid growth of conversational ai applications. to design a natural language understanding system, a set of expected customer intentions are collected beforehand to train an intent recognition model. however, the pre-defined intents cannot fully meet customer needs. this implies the necessity of expanding the intent recognition model by repeatedly integrating new intents discovered from unlabeled user utterances (fig. 1 ). to reduce the effort in manually identifying unknown intents from a mass of utterances, previous works commonly employ clustering algorithms to group utterances of similar intents @xcite @xcite . the cluster assignments thereafter can either be directly used as new intent labels or as heuristics for faster annotations. research questions (rq) and challenges. current study of nid centers around two basic research questions: 1) how to learn semantic utterance representations to provide proper cues for clustering? 2) how to better cluster the utterances? the study of the two questions are often interwoven in existing research. utterances can be represented according to different aspects such as the style of language, the related topics, or even the length of sentences. it is important to learn semantic utterance representations to provide proper cues for clustering. simply applying a vanilla pre-trained language model (plm) to generate utterance representations is not a viable solution, which leads to poor performance on nid as shown by the experimental results in section 4.2. some recent works proposed to use labeled utterances of known intents for representation learning @xcite @xcite @xcite , but they require a substantial amount of known intents and sufficient labeled utterances of each intent, which are not always available especially at the early development stage of a dialogue system. further, pseudo-labeling approaches are often exploited to generate supervision signals for representation learning and clustering. for example, @xcite finetune a plm with an utterance similarity prediction task on labeled utterances to guide the training of unlabeled data with pseudo-labels. @xcite adopt a deep clustering method @xcite that uses k-means clustering to produce pseudo-labels. however, pseudo-labels are often noisy and can lead to error propagation. our solutions. in this work, we propose a simple yet effective solution for each research question. solution to rq 1: multi-task pre-training. we propose a multi-task pre-training strategy that takes advantage of both external data and internal data for representation learning. specifically, we leverage publicly available, high-quality intent detection datasets, following @xcite , as well as the provided labeled and unlabeled utterances in the current domain, to fine-tune a pre-trained plm to learn task-specific utterance representations for nid. the multi-task learning strategy enables knowledge transfer from general intent detection tasks and adaptation to a specific application domain. solution to rq 2: contrastive learning with nearest neighbors. we propose to use a contrastive loss to produce compact clusters, which is motivated by the recent success of contrastive learning in both computer vision @xcite @xcite and natural language processing @xcite @xcite . contrastive learning usually maximizes the agreement between different views of the same example and minimize that between different examples. however, the commonly used instance discrimination task may push away false negatives and hurts the clustering performance. inspired by a recent work in computer vision @xcite , we introduce neighborhood relationship to customize the contrastive loss for clustering in both unsupervised (i.e., without any labeled utterances of known intents) and semi-supervised scenarios. intuitively, in a semantic feature space, neighboring utterances should have a similar intent, and pulling together neighboring samples makes clusters more compact. our main contributions are three-fold. to tackle semi-supervised nid, lin et al. (2020) proposed to first perform supervised training on known intents with a sentence similarity task and then use pseudo labeling on unlabeled utterances to learn a better embedding space. @xcite proposed to first pre-train on known intents and then perform k-means clustering to assign pseudo labels on unlabeled data for representation learning following deep clustering @xcite . they also proposed to align clusters to accelerate the learning of top layers. another approach is to first classify the utterances as known and unknown and then uncover new intents with the unknown utterances @xcite . hence, it relies on accurate classification in the first stage. in this work, we address nid by proposing a multi-task pre-training method for representation learning and a contrastive learning method for clustering. in contrast to previous methods that rely on ample annotated data in the current domain for pre-training, our method can be used in an unsupervised setting and work well in data-scarce scenarios (section 4.3). pre-training for intent recognition. despite the effectiveness of large-scale pre-trained language models @xcite @xcite , the inherent mismatch in linguistic behavior between the pre-training datasets and dialogues encourages the research of continual pre-training on dialogue corpus. most previous works proposed to pre-train on open domain dialogues in a selfsupervised manner @xcite @xcite . recently, several works pointed out that pre-training with relavant tasks can be effective for intent recognition. for example, @xcite formulated intent recognition as a sentence similarity task and pre-trained on natural language inference (nli) datasets. @xcite ; zhang et al. (2021e) pre-trained with a contrastive loss on intent detection tasks. our multi-task pre-training method is inspired from @xcite which leverages publicly available intent datasets and unlabeled data in the current domain for pre-training to improve the performance of few-shot intent detection. however, we argue that the method is more suitable to be applied for nid due to the natural existence of unlabeled utterances. contrastive representation learning. contrastive learning has shown promising results in computer vision @xcite @xcite and gained popularity in natural language processing. some recent works used unsupervised contrastive learning to learn sentence embeddings @xcite @xcite . specifically, @xcite ; @xcite showed that contrastive loss can avoid an anisotropic embedding space. @xcite proposed a self-guided contrastive training to improve the quality of bert representations. @xcite proposed to pre-train a universal sentence encoder by contrasting a randomly sampled text segment from nearby sentences. @xcite demonstrated that self-supervised contrastive pre-training and supervised contrastive fine-tuning can benefit few-shot intent recognition. @xcite showed that combining a contrastive loss with a clustering objective can improve short text clustering. our proposed contrastive loss is tailored for clustering, which encourages utterances with similar semantics to group together and avoids pushing away false negatives as in the conventional contrastive loss. overview of our approach. as shown in fig. 2 , we propose a two-stage framework that addresses the research questions mentioned in sec. 1. in the first stage, we perform multi-task pre-training (mtp) that jointly optimizes a crossentropy loss on external labeled data and a selfsupervised loss on target unlabeled data (sec. 3.1). in the second stage, we first mine top-k nearest neighbors of each training instance in the embedding space and then perform contrastive learning with nearest neighbors (clnn) (sec. 3.2). after training, we employ a simple non-parametric clustering algorithm to obtain clustering results. ## conclusion we have provided simple and effective solutions for two fundamental research questions for new intent discovery (nid): (1) how to learn better utterance representations to provide proper cues for clustering and (2) how to better cluster utterances in the representation space. in the first stage, we use a multi-task pre-training strategy to exploit both external and internal data for representation learning. in the second stage, we perform contrastive learning with mined nearest neighbors to exploit self-supervisory signals in the representation space. extensive experiments on three intent recognition benchmarks show that our approach can significantly improve the performance of nid in both unsupervised and semi-supervised scenarios. there are two limitations of this work. (1) we have only evaluated on balanced data. however, in real-world applications, most datasets are highly imbalanced. (2) the discovered clusters lack interpretability. our clustering method can only assign a cluster label to each unlabeled utterance but cannot generate a valid intent name for each cluster.