Learning Geometry-Aware Representations for New Intent Discovery (2024)

Abstract: New intent discovery (NID) is an important problem for deploying practical dialogue systems, which trains intent classifiers on a semi-supervised corpus where unlabeled user utterances contain both known and novel intents. Most existing NID algorithms place hope on the sample similarity to cluster unlabeled corpus to known or new samples. Lacking supervision on new intents, we experimentally find the intent classifier fails to fully distinguish new intents since they tend to assemble into intertwined centers.To address this problem, we propose a novel GeoID framework that learns geometry-aware representations to maximally separate all intents. Specifically, we are motivated by the recent findings on Neural Collapse (NC) in classification tasks to derive optimal intent center structure. Meanwhile, we devise a dual pseudo-labeling strategy based on optimal transport assignments and semi-supervised clustering, ensuring proper utterances-to-center arrangement.Extensive results show that our GeoID method establishes a new state-of-the-art performance, achieving a +3.49% average accuracy improvement on three standardized benchmarking datasets. We also verify its usefulness in assisting large language models for improved in-context performance.

## introduction intent detection, which aims at recognizing the intention of the user query, is an important component in industrial-grade dialogue systems @xcite . though there has been significant progress in recent years @xcite @xcite , the effectiveness of these models greatly depends on the predefined intent labels, which often fall short of fulfilling the application requirements in * corresponding author. figure 1 : illustration for the effect of our off-line etf structure: having gotten suboptimal features from the simple similarity-based method, we achieve an optimal feature distribution structure by bringing the samples closer to a fixed etf structure. an open-world environment. to address this problem, new intent discovery (nid) has been proposed that learns from a semi-supervised corpus containing: (i)-a limited amount of utterance annotated with true intents; (ii)-a vast quantity of unlabeled data containing both known intents and unknown new intents. the goal of nid is to simultaneously classify known intents while discovering those unknown intent clusters. emerging attempts have been made to tackle the nid task @xcite @xcite . amongst them, the most popular methods @xcite rely on the similarity-based assumption, i.e., samples sharing the same intent typically located at a neighbor region. then, they design specific objectives to spontaneously group different samples into separated clusters, e.g. pairwise constraints @xcite , contrastive learning @xcite , and graph diffusion @xcite . however, without new intent supervision, the learning procedure can be dominated by the labeled samples, resulting in biased feature representations. thus, the unknown samples are prone to collapse into intertwined centers, thus being inseparable, as empirically verified in figure 5 . this gives rise to a representation dilemma-how to obtain the optimal hypersphere distribution for identifying new intents? to answer this question, we draw inspiration from a recent study on the neural collapse (nc) phenomenon @xcite . it implies that, under ideal conditions, as a classifier undergoes training towards convergence, the final-layer characteristics of a specific class gradually converge towards their average value within that class. these averages then tend to position themselves on a hypersphere with the utmost angular distance, known as the simplex equiangular tight frame (simplex etf), as referenced in @xcite . the nc phenomenon provides an optimal frame of hypersphere centers that aligns with our expectations for the final intent centers to be maximally separated. nevertheless, it is non-trivial to achieve nc for nid with largely missing supervision. in this paper, we propose a novel geometryaware representation learning framework for nid (dubbed geoid), which encourages both known/unknown intents to be equiangular separative and maximally discriminative. specifically, geoid replaces the learnable weights of the intent classifier (implicitly the class centers) with fixed etf weights. that is, it pulls the sample features to predefined intent centers instead of merely clustering by sample similarity themselves. moreover, we develop a dual pseudo-labeling strategy to ensure proper sample allocation to the centers: (i)optimal transport-based objective to refine pseudolabels from classifier predictions, which is reliable with pattern fitting; (ii)-semi-supervised clusteringdriven pseudo-labeling that is more robust in the early stages. finally, we train the nid classifier to fit both pseudo-labels for improved robustness. we comprehensively evaluate geoid on three nid benchmark datasets. specifically, our visualized analysis shows that geoid does indeed learn highly distinguishable intent clusters (figure 5(c )), and its mean intent centers almost achieve neural collapsed distribution (figure 3(c) ). thanks to this excellent geometry property, our geoid establishes new state-of-the-art nid performance, e.g., it improves the accuracy by +2.90%, +3.88%, and +3.69% accuracy on the banking, stackoverflow, and clinc datasets, respectively. we hope our work will inspire future nid studies to tackle this important problem of cluster separability. neural collapse (nc). the nc phenomenon was initially discovered by @xcite . they observed that towards the end of training, a classification model would exhibit the collapse of last-layer features towards their respective withinclass centers. these within-class centers, along with the classifiers, will converge to form a simplex equiangular tight frame (etf). subsequent studies have sought to provide theoretical insights into this elegant phenomenon. it has been proven that neural collapse corresponds to the global optimality of simplified models under various conditions, including regularization @xcite @xcite and constraints @xcite . these findings hold for both cross-entropy (ce) @xcite and mean squared error (mse) loss functions @xcite . phenomenon of neural collapse (nc) has also been explored in specific scenarios, such as imbalanced learning @xcite @xcite , learning with noisy labels @xcite , and transfer learning @xcite ), but they have never tapped in language domain. we are the first to investigate the nc phenomenon in the nid task. intent representation. similar to the majority of mainstream tasks, we use bert @xcite to extract intent representations. first, we input utterance x i to bert and get all its token embeddings [cls, t 1 , . . . , t m ]. next, we utilize mean-pooling and dense layer to obtain intent feature representation z i . for contrastive learning, we utilize random token replacement (rtr) following @xcite as data augmentation strategy to generate z i ′ . overview of our approach. figure 2 illustrates the overall architecture of our method. we learn geometry-aware representations to maximally separate all intents by pushing utterances close to the corresponding vertex of the etf structure inspired by the nc phenomenon. to achieve this, we devise pseudo-labeling a dual strategy based on optimal transport and clustering to assign samples to the correct centers. ## conclusion in this paper, we propose a new method geoid which revisits new intent discovery from the perspective of geometric-aware representation learning. inspired by the neural collapse phenomenon, we use a fixed etf structure to facilitate the learning of a feature distribution that aligns with the predefined optimal structure. additionally, we design a dual pseudo-labeling strategy using optimal transport theory and semi-supervised clustering to generate pseudo-labels respectively. this strategy enhances the quality of pseudo-labels while maintaining sample utilization. experiments show geoid's significant improvements over baselines. we hope our work can inspire future research to further improve the nid problem from the perspective of geometry-aware representations.