Towards Real-world Scenario: Imbalanced New Intent Discovery (2024)

Abstract: New Intent Discovery (NID) aims at detecting known and previously undefined categories of user intent by utilizing limited labeled and massive unlabeled data. Most prior works often operate under the unrealistic assumption that the distribution of both familiar and new intent classes is uniform, overlooking the skewed and long-tailed distributions frequently encountered in real-world scenarios. To bridge the gap, our work introduces the imbalanced new intent discovery i-NID task, which seeks to identify familiar and novel intent categories within long-tailed distributions. A new benchmark baNID-Bench comprised of three datasets is created to simulate the real-world long-tail distributions. ImbaNID-Bench ranges from broad cross-domain to specific single-domain intent categories, providing a thorough representation of practical use cases. Besides, a robust baseline model ImbaNID is proposed to achieve cluster-friendly intent representations. It includes three stages: model pre-training, generation of reliable pseudo-labels, and robust representation learning that strengthens the model performance to handle the intricacies of real-world data distributions. Our extensive experiments on previous benchmarks and the newly established benchmark demonstrate the superior performance of ImbaNID in addressing the i-NID task, highlighting its potential as a powerful baseline for uncovering and categorizing user intents in imbalanced and long-tailed distributions.

## introduction new intent discovery (nid) has captured increasing attention due to its adaptability to the evolving user needs in open-world scenarios @xcite @xcite @xcite . nid methods generally follow a two-stage training process, including a knowledge transfer and a discovery stage. the prior knowledge is injected into the model via pre-training and then the discriminative representation is learned for known and novel intent categories @xcite @xcite @xcite . despite the considerable advancements in nid, there remain two salient challenges impeding adoption in practical scenarios. in fig. 1 , most nid approaches predominantly address the issue of intent discovery within the framework of balanced datasets. but the distribution of intents often follows a long-tailed pattern @xcite , particularly in dialogue systems, wherein a small number of intents are highly represented and a wide variety of intents (unknown intents) are sparsely exemplified. secondly, nid methods suffer from severe clustering degradation, where lack of improved methods for unbalanced data distributions and leading to poor performance in unbalanced scenarios. therefore, we explore the new methods under the imbalanced new intent discovery (i-nid) task to bridge the gap between the nid and real-world applications. to break out the aforementioned limitations, we propose a novel framework imbanid, which includes three key components: model pre-training, reliable pseudo-labeling (rpl), and robust representation learning (rrl). specifically, the multitask pre-training incorporates the generalized prior knowledge into the mode for establishing a robust representational foundation conducive to clustering known and novel intents. the rpl component formulates the pseudo-label generation as a relaxed optimal transport problem, applying adaptive constraints to recalibrate the class distribution for enhanced uniformity. the model bias issues can be mitigated in long-tail settings while furnishing reliable supervisory cues for downstream representation learning. then, a novel distribution-aware and quality-aware noise regularization technique is introduced in rrl to effectively distinguish between clean and noisy samples. a contrastive loss function is subsequently used to facilitate the formation of distinct and well-separated clusters of representations for known and novel intent categories. the collaborative synergy between rpl and rrl fosters an iterative training process to create a symbiotic relationship. this iterative approach cultivates intent representations conducive to clustering, significantly aiding the i-nid task. for better evaluation of unbalanced distribution, we introduce a comprehensive benchmark imbanid-bench for i-nid evaluation. extensive experiments of imbanid are evaluated on the previous common benchmarks and our proposed benchmark imbanid-bench. the results demonstrate that imbanid consistently achieves state-of-the-art performance across all clusters, notably surpassing standard nid models by an average margin of 2.7% in long-tailed scenarios. the contributions are summarized as follows: ## datasets we introduce a new benchmark (called imbanid-bench) for nid evaluation tailored to long-tail distribution scenarios, which comprises three datasets named clinc150-lt, banking77-lt, and stackoverflow20-lt, derived from clinc @xcite , banking @xcite and stackoverflow @xcite . comprehensive statistics for each dataset are documented in appendix b. here, we describe the details of the imbanid-bench datasets. ## conclusion in this work, we first propose the i-nid task to identify known and infer novel intents within these long-tailed distributions. then, we develop an effective imbanid baseline method for the i-nid task, where pseudo-label generation and representation learning mutually iterate to achieve cluster-friendly representations. comprehensive experimental results on our imbanid-bench benchmark datasets demonstrate the effectiveness of our imbanid method for i-nid. we hope our work will draw more attention from the community toward a broader view of tackling the i-nid problem. initially, we fix β and h, and then update q, f , and g. by setting ∂l ∂q ij , ∂l ∂f i , and ∂l ∂g j to zero, we obtain the following results: based on eq. ( 20 ), we derive the following: considering the constraints (21) and the conditions β t 1 = α t 1 = 1, we solve eq. ( 22 ) to determine the values of q, f , and g using the sinkhorn algorithm @xcite . subsequently, with f , g, and q fixed, we update β and h. setting eq. ( 18 ) to zero yields the following solution: take eq. ( 23 ) into the eq. ( 19 ) and let eq. ( 19 ) equal to 0, we can obtain: algorithm 1 the optimization of rot input: the cost matrix:log p. output: the transport matrix: q, the class distribution: β. procedure: we obtain h from eq.( 24 ) using the bisection method and subsequently determine the corresponding β. in the final step, we iteratively update f , g, q, and β, h. the iterative optimization process for rot is outlined in algo-rithm1.