A Diffusion Weighted Graph Framework for New Intent Discovery (2023)

Abstract: New Intent Discovery (NID) aims to recognize both new and known intents from unlabeled data with the aid of limited labeled data containing only known intents. Without considering structure relationships between samples, previous methods generate noisy supervisory signals which cannot strike a balance between quantity and quality, hindering the formation of new intent clusters and effective transfer of the pre-training knowledge. To mitigate this limitation, we propose a novel \textit{Diffusion Weighted Graph Framework} (DWGF) to capture both semantic similarities and structure relationships inherent in data, enabling more sufficient and reliable supervisory signals. Specifically, for each sample, we diffuse neighborhood relationships along semantic paths guided by the nearest neighbors for multiple hops to characterize its local structure discriminately. Then, we sample its positive keys and weigh them based on semantic similarities and local structures for contrastive learning. During inference, we further propose \textit{Graph Smoothing Filter} (GSF) to explicitly utilize the structure relationships to filter high-frequency noise embodied in semantically ambiguous samples on the cluster boundary. Extensive experiments show that our method outperforms state-of-the-art models on all evaluation metrics across multiple benchmark datasets. Code and data will be made public.

## introduction even though current machine learning methods have achieved superior performance on many nlp tasks, they often fail to meet application requirements in an open-world environment. for instance, general intent classification models trained on predefined intents cannot recognize new intents from unlabeled dialogues, which is a clear obstacle for real-world applications. therefore, research on new intent discovery (nid), which aims to discover new intents from unlabeled data automatically, has attracted much attention recently. most existing nid methods @xcite @xcite @xcite adopt a two-stage training strategy: pre-training on labeled data, then learning clustering-friendly representation with pseudo supervisory signals. however, previous methods only rely on semantic similarities to generate supervisory signals based on the assumption that samples within the feature hypersphere belong to the same category as the hypersphere anchor, e.g. cluster centroids @xcite , class prototypes @xcite , or query samples @xcite . even though these methods can learn some discriminative features, they still face limitations in generating both adequate and reliable supervisory signals, which we call the quantity and quality dilemma. specifically, as shown in fig. 1 bottom left, these methods rely on a fixed threshold to determine the search radius of the hypersphere. shrinking the threshold (blue solid line) helps retrieve more accurate positive keys, but it loses information from positive keys out of the hypersphere, resulting in a low recall. however, simply relaxing the threshold (red dashed line) will introduce much noise and lead to low accuracy. quantity and quality dilemma is caused by the fact that the previous methods searched positive keys indiscriminately along all directions of the hypersphere with a fixed search radius. in order to selectively sample both adequate and reliable positive keys to ensure the formation of new intent clusters, we propose to model and utilize structure relationships inherent in data, which reflect the semantic correlations between samples from the perspective of connectivity. as shown in fig. 1 top, for each sample, we first initialize its k-nearest neighbors with a tightened threshold. then we connect any two samples if they have at least one shared neighbor since the semantics of the shared neighbor are highly correlated with the samples on both sides. according to this rule, we identify two samples (with brown borders in fig. 1 top) that can be used as bridges and diffuse the anchor along them to search positive keys near the boundary of the hypersphere, forming the final semantic path. in the case of the same semantic similarity, we additionally require the positive keys to appear on the semantic paths diffused from the anchor. in this paper, we propose a novel diffusion weighted graph framework to model and utilize structure relationships. specifically, from any anchor, we diffuse neighborhood relationships along the nearest neighbor-guided semantic paths for multiple hops to construct the final dwg. as shown in fig. 1 bottom right, then we sample positive keys along the semantic paths (arrow lines) in dwg within the relaxed feature hypersphere. moreover, sampled keys are assigned to different contrastive weights according to their frequency of being sampled on different semantic paths, where keys that are diffused repeatedly from different outsets will accumulate larger values and vice versa. we conduct contrastive learning with sampled positive keys and corresponding weights in the embedding space. apart from considering the sample-sample structure relationships from the local view, we adopt the idea of @xcite to help learn clustering-friendly representations from the global view through self-training. during the inference stage, in order to filter highfrequency noise embodied in the semantically ambiguous samples on the cluster boundary, we propose a novel inference improvement graph smoothing filter (gsf), which utilizes normalized graph laplacian to aggregate neighborhood information revealed by structure relationships of testing samples. smoothed testing features help to obtain better clustering results. our main contributions can be summarized as follows: ## conclusion in this paper, we propose a novel diffusion weighted graph framework (dwgf) for new intent discovery, which models structure relationships inherent in data through nearest neighborguided diffusion. combined with structure relationships, we improve both the sampling and weighting strategy in contrastive learning and adopt supervision from local and global views. we further propose graph smoothing filter (gsf) to explore the potential of structure relationships in inference, which effectively filters noise embodied in semantically ambiguous samples on the cluster boundary. extensive experiments on all three clustering metrics across multiple benchmark datasets fully validate the effectiveness and robustness of our method.