Continual Generalized Intent Discovery: Marching Towards Dynamic and Open-world Intent Recognition (2023)

Abstract: In a practical dialogue system, users may input out-of-domain (OOD) queries. The Generalized Intent Discovery (GID) task aims to discover OOD intents from OOD queries and extend them to the in-domain (IND) classifier. However, GID only considers one stage of OOD learning, and needs to utilize the data in all previous stages for joint training, which limits its wide application in reality. In this paper, we introduce a new task, Continual Generalized Intent Discovery (CGID), which aims to continuously and automatically discover OOD intents from dynamic OOD data streams and then incrementally add them to the classifier with almost no previous data, thus moving towards dynamic intent recognition in an open world. Next, we propose a method called Prototype-guided Learning with Replay and Distillation (PLRD) for CGID, which bootstraps new intent discovery through class prototypes and balances new and old intents through data replay and feature distillation. Finally, we conduct detailed experiments and analysis to verify the effectiveness of PLRD and understand the key challenges of CGID for future research.

## introduction the traditional intent classification (ic) in a taskoriented dialogue system (tod) is based on a closed set assumption @xcite @xcite and can only handle queries within a limited scope of in-domain (ind) intents. however, users may input out-of-domain (ood) queries in the real open world. recently, the research community has paid more attention to ood problems. ood detection @xcite @xcite aims to identify whether a user's query is outside the range of the predefined intent set to avoid wrong operations. it can safely reject ood intents, but it also ignores ood concepts that are valuable for future development. ood intent discovery @xcite @xcite helps determine potential development directions by grouping unlabelled ood data into different clusters, but still cannot incrementally expand the recognition scope of existing ind classifiers. generalized intent discovery (gid) @xcite further trains a network that can classify a set of labelled ind intent classes and simultaneously discover new classes from an unlabelled ood set and incrementally add them to the classifier. although gid realizes the incremental expansion of the recognition scope of the intent classifier without any new intents labels, two major problems limit the widespread application of gid in reality as shown in fig 1 : (1) gid only considers singlestage of ood discovery and classifier expansion. in real scenarios, ood data is gradually collected over time. even if the current intent classifier is incrementally expanded, new ood queries and intents will continue to emerge. besides, the timeliness of ood discovery needs to be considered: timely discovery of new intents and expansion to the system can help improve the subsequent user experience. (2) gid require data in all previous stages for joint training to maintain the classification ability for known intents. since ood samples are collected from users' real queries, storing past data may bring serious privacy issues. in addition, unlike class incremental learning (cil) that require new classes with real labels, it is hard to obtain a large amount of dynamic labeled data in reality, and the label set for ood queries is not predefined and needs to be mined from query logs. inspired by the above problems, in this paper, we introduce a new task, continual generalized intent discovery (cgid), which aims to continually and automatically discover ood intents from ood data streams and expand them to the existing ind classifier. in addition, cgid requires the system to maintain the ability to classify known intents with almost no need to store previous data, which makes existing gid methods fails to be applied to cgid. through cgid, the ic system can continually enhance the ability of intent recognition from unlabeled ood data streams, thus moving towards dynamic intent recognition in an open world. we show the difference between cgid and gid, as well as the cil task in fig 2 and then leave the definition and evaluation protocol in section 2. as cgid needs to continuously learn from unlabeled ood data, it is foreseeable that the system will inevitably suffer from the catastrophic forgetting @xcite of known knowledge as well as the interference and propagation of ood noise @xcite . to address the issues, we propose the prototype-guided learning with replay and distillation (plrd) for cgid. specifically, plrd consists of a main module composed of an encoder and a joint classifier, as well as three sub-modules: (1) class prototype guides pseudo-labels for new ood samples and alleviate the ood noise; (2) feature distillation reduces catastrophic forgetting; (3) a memory balances new class learning and old class classification by replaying old class samples (section 3) foot_0 . furthermore, to verify the effectiveness of plrd, we construct two public datasets and three baseline methods for cgid. extensive experiments prove that plrd has significant performance improvement and the least forgetting compared to the baselines, and achieves a good balance among old classes classification, new class discovery and incremental learning (section 4). to further shed light on the unique challenges faced by the cgid task, we conduct detailed qualitative analysis (section 5). we find that the main challenges of cgid are conflicts between different sub-tasks, ood noise propagation, fine-grained ood classes and strategies for replayed samples (section 6), which provide profound guidance for future work. our contributions are three-fold: (1) we introduce a new task, continual generalized intent discovery (cgid), to achieve the dynamic and openworld intent recognition and then construct datasets and baselines for evaluating cgid. (2) we propose a practical method plrd for cgid, which guides new samples through class prototypes and balances new and old tasks through data replay and feature distillation. (3) we conduct comprehensive experiments and in-depth analysis to verify the effectiveness of plrd and understand the key challenges of cgid for future work. ## problem definition in this section, we first briefly introduce the generalized intent discovery(gid) task, then delve into the details of the continual generalized intent discovery (cgid) task we proposed. bert frozen bert joint classifier prototype pseudo labels new samples replay samples bert ind classifier push away pull together old samples new samples new prototypes old prototypes pull together copy replayed labels ## challenges based on the above experiments and analysis, we comprehensively summarize the unique challenges faced by cgid : conflicts between different sub-tasks in cgid, the discovery and classification of new ood classes tend towards different features, and learning new ood classes interfere with existing knowledge about old classes inevitably. however, preventing forgetting will lead to model rigidity and is not conducive to the learning of new classes. ## conclusion in this paper, we introduce a more challenging yet practical task as continuous generalized intent discovery (cgid), which aims at continuously and automatically discovering new intents from ood data streams and incrementally extending the classifier, thereby enabling dynamic intent recognition in an open-world setting. to address this task, we propose a new method called prototype-guided learning with replay and distillation (plrd). extensive experiments and qualitative analyses validate the effectiveness of plrd and provide insights into the key challenges of cgid.