Pseudo-Label Enhanced Prototypical Contrastive Learning for Uniformed Intent Discovery (2024)

Abstract: New intent discovery is a crucial capability for task-oriented dialogue systems. Existing methods focus on transferring in-domain (IND) prior knowledge to out-of-domain (OOD) data through pre-training and clustering stages. They either handle the two processes in a pipeline manner, which exhibits a gap between intent representation and clustering process or use typical contrastive clustering that overlooks the potential supervised signals from the whole data. Besides, they often deal with either open intent discovery or OOD settings individually. To this end, we propose a Pseudo-Label enhanced Prototypical Contrastive Learning (PLPCL) model for uniformed intent discovery. We iteratively utilize pseudo-labels to explore potential positive/negative samples for contrastive learning and bridge the gap between representation and clustering. To enable better knowledge transfer, we design a prototype learning method integrating the supervised and pseudo signals from IND and OOD samples. In addition, our method has been proven effective in two different settings of discovering new intents. Experiments on three benchmark datasets and two task settings demonstrate the effectiveness of our approach.

## introduction task-oriented conversational systems are typically designed to assist users in completing specific tasks by leveraging predefined ontologies (e.g., intents, slots and values), which face limitations in dynamic environments where novel ontology categories may emerge @xcite @xcite @xcite . new intent discovery, which aims to uncover and categorize out-of-domain intents absent from the training data, has received increasing attention due to its crucial role in dialogue systems @xcite . initially, researchers focused on exploring unsupervised clustering methods @xcite @xcite . however, real-world scenarios often involve limited labeled data, prompting a shift toward semi-supervised approaches, notably ood and open intent discovery @xcite @xcite @xcite @xcite . ood intent discovery involves clustering unlabeled ood intents by utilizing labeled ind data, focusing solely on identifying novel categories absent in the training set. in contrast, open intent discovery seeks to simultaneously recognize both known and new categories from unlabeled data, allowing for a more comprehensive understanding of both in-domain and out-of-domain intents (see figure 1 ). the existing approaches for new intent discovery typically use a two-stage contrastive clustering approach, involving ind pre-training and ood clustering. for open intent discovery, researchers focus on effectively utilizing the small amount of labeled data for weakly supervised @xcite or semi-supervised clustering methods @xcite . for ood intent discovery, previous works commonly employ a contrastive clustering framework @xcite with approaches such as multi-head contrast learning @xcite or neighbor-enhanced contrastive strategies @xcite . although previous methods in intent discovery have achieved notable success, several challenges in the field remain unexplored. (1) one key challenge is efficiently integrating labeled and unlabeled data for joint representation learning and clustering. while some methods focus on contrastive clustering for joint learning @xcite , they often overlook critical supervised signals from ind samples during the clustering stage. leveraging labeled information for this process remains underexplored, particularly given the limited availability of labeled data and the risk of overfitting. (2) another critical issue is devising effective transfer learning mechanisms between ind and ood data while preventing catastrophic forgetting. existing methods often discard classifiers trained on prior knowledge, retaining only feature extraction during clustering with ood samples @xcite . this requires additional alignment strategies, potentially introducing noise if suboptimal. there's an urgent need for methods preserving prior knowledge while adapting better to new intent data, ensuring seamless transfer learning. (3) moreover, the predominant focus of prior research has been either on open intent discovery or ood intent discovery individually, disregarding the practical need for a unified method capable of handling both scenarios @xcite . real-world dialogue systems often encounter situations requiring updates or migration, underscoring the need for a uniform intent discovery approach that can adapt to various system changes. to address these limitations, we propose a pseudo-label enhanced protypical contrastive learning (plpcl) model which is built upon contrastive clustering for joint representation learning and clustering. our approach begins by pretraining the contrastive clustering model using labeled ind data. to effectively harness labeled information, we integrate ind samples with unlabeled data using a semi-supervised clustering strategy, employing distinct contrastive learning strategies for labeled and unlabeled data. to prevent overfitting and maximize the utilization of unlabeled data, we iteratively select reliable unlabeled samples with confident pseudo-labels. these reliable samples serve as potential positive/negative samples during contrastive learning, enhancing the overall contrastive clustering process. to bridge the gap between ind and ood data, we introduce a prototype learning strategy. it maintains the prototype matrix by integrating instance and cluster features from both the ind and reliable unlabeled samples. our method integrates contrastive clustering and prototypical learning, eliminating the need for an extra aligning module. this integration facilitates improved knowledge transfer from ind to ood without discarding label information from ind samples. furthermore, our framework is purposefully designed for uniform intent discovery, demonstrating effectiveness in both open-setting and ood-setting scenarios. the contribution of our work is threefold: (1) we introduce a novel method that leverages both labeled and unlabeled data through pseudo-label enhanced semi-supervised contrastive learning. this approach facilitates joint representation learning and clustering by effectively leveraging the whole data. ( 2 ) we propose a prototypical contrastive learning model for uniformed intent discovery integrating prototypical learning and contrastive clustering to bridge the gap between ind prior knowledge and ood categories. (3) we conduct extensive experiments on both ood and open intent discovery scenarios and the results demonstrate the effectiveness of our proposed method. 2 related work 2.1 intent discovery recent research for intent discovery can be broadly categorized into ood-setting and open-setting. as shown in figure 1, open intent discovery involves clustering both ind and ood intents with ind priori knowledge. samples with ind intents are not all labeled. whereas ood intent discovery focuses on accurately handling ood intents and assumes that the intents of labeled and unlabeled data do not overlap, which means all ind samples are labeled. lin et al. (2020) proposed a self-supervised clustering method that utilizes limited labeled data. zhang et al. (2021b) proposed a k-means-based semi-supervised clustering method that can effectively use prior knowledge in intent discovery. mou et al. (2022c) proposed a disentangled contrastive learning framework that mainly focuses on ood intent clustering and decouples instance and cluster-joint representation and cluster prior knowledge retention reuse of labeled data ood setting open setting cdac+ (lin et al., 2020) ✓ pair-wise similarity ✓ × ✓ deepaligned (zhang et al., 2021b) × representation × × ✓ dkt (mou et al., 2022c) ✓ representation × ✓ × dpn (an et al., 2023) × prototype ✓ × ✓ plpcl (ours) ✓ representation, prototype, classifier ✓ ✓ ✓ ## conclusion in this paper, we propose a pseudo-label enhanced prototypical contrastive learning approach for both open intent discovery and ood intent discovery. the pseudo-label filtering strategy enhances supervised signal during the training process, while the prototypical contrastive learning module addresses the isolation issue between two independent contrastive learning heads. experiments on two task settings and three benchmark datasets demonstrate the effectiveness of our proposed method. we hope to explore more self-supervised methods for ood and open intent discovery in the future.