Generalized Intent Discovery: Learning from Open World Dialogue System (2022)

Abstract: Traditional intent classification models are based on a pre-defined intent set and only recognize limited in-domain (IND) intent classes. But users may input out-of-domain (OOD) queries in a practical dialogue system. Such OOD queries can provide directions for future improvement. In this paper, we define a new task, Generalized Intent Discovery (GID), which aims to extend an IND intent classifier to an open-world intent set including IND and OOD intents. We hope to simultaneously classify a set of labeled IND intent classes while discovering and recognizing new unlabeled OOD types incrementally. We construct three public datasets for different application scenarios and propose two kinds of frameworks, pipeline-based and end-to-end for future work. Further, we conduct exhaustive experiments and qualitative analysis to comprehend key challenges and provide new guidance for future GID research.

## introduction intent classification (ic) in a dialogue system aims to identify the goal of a user query, such as book-flight or addtoplaylist. recent neural-based models @xcite @xcite @xcite have achieved satisfying performance under the availability of large-scale labeled data. however, these methods face the challenge of data scarcity and poor scalability. they rely on a pre-defined intent set and supervised labels, which is limitted in some practical scenarios. existing intent classification models have little to offer in an open-world setting, in which many new intent categories are not defined apriori and no labeled data is available. these models rely on the pre-defined intent set, making it only recognize limited in-domain (ind) intent categories. but plenty of input queries may be outside of the fixed intent set, which we call out-of-domain (ood) intents @xcite . in recent years, ood intent detection @xcite @xcite @xcite has been well studied, which identifies whether a user query falls outside the range of pre-defined intent set to avoid performing wrong operations. but it can only safely reject ood intents thus ignore these valuable ood concepts for future development. further, ood intent discovery task (also known as new intent discovery) @xcite ) is proposed to cluster unlabeled ood data. the adopted clustering method can only group those ood intents into clusters, but cannot further expand the recognition scope of the existing ind intent classifier incrementally. inspired by the above issues, we introduce a new task of extending and recognizing intent categories automatically, generalized intent discovery(gid). gid aims to extend an existing ind intent classifier to an open-world ood intent set, as shown in fig 1 . the main motivation is that we hope to train a network that can simultaneously classify a set of labeled ind intent classes while discovering new ones in an unlabeled ood set. in this way, we can enhance the capability of an ic system by expanding its recognition scope incrementally. we show a comparison of gid and existing ood tasks in fig 2 . since the practical ood intents are unsupervised, neither the ood labels nor ood intent schema make it different from zero-shot learning @xcite and continual learning @xcite which both rely on a given label ontology, like label descriptions. therefore, to explore unique characteristics of gid, we construct three kinds of gid benchmarks, including single domain, multiple domain, and crossdomain settings (section 3). these settings denote different application scenarios which we will discuss later. subsequently, we propose two kinds of frameworks for gid, pipeline and end-to-end. a straightforward idea is pipeline-based methods which firstly learn ood cluster assignments and get pseudo ood labels, then jointly classify labeled ind data and pseudo labeled ood data. however, pipeline-based methods separate ood clustering and classification process, which ignores the interaction between labeled ind data and unlabeled ood data. besides, these pseudo ood labels may induce severe noise to the joint classification, limiting the performance of the joint ind and ood classifiers. therefore, we further propose an endto-end framework to simultaneously learn pseudo ood cluster labels and classify ind&ood classes along with ground truth ind labels via a unified objective. we obtain the pseudo label of an ood query by its augmented view in a swapped prediction way @xcite @xcite and employ the sinkhorn-knopp (sk) algorithm @xcite to solve the optimization problem. we leave the details to section 4. we also perform exhaustive experiments (section 5.2) and qualitative analysis (section 5.3) to shed light on the challenges that current approaches face with gid. we find fine-grained ood types, domain gap, data imbalance, real ood noise and estimating the number of ood types are the main challenges (section 6), which provide insightful guidance for future gid work. our contributions are four-fold: (1) we introduce a new task, generalized intent discovery (gid) which aims to extend an ind intent classifier to an open-world ood intent set. gid helps expand the model's recognition scope and develop new skills for improving dialogue systems. (2) we construct three kinds of public gid benchmarks for different application scenarios, which help to explore the key challenges of gid comprehensively. ## problem formulation in this section, we first briefly introduce the traditional intent classification (ic) task, then dive into the details of our proposed generalized intent discovery (gid) task. ## dataset to explore the practical significance and key challenges of gid task, we need to construct the gid dataset. however, we found that in some related tasks such as ood intent discovery @xcite and zero-shot intent detection @xcite , the commonly used construction methods are to randomly divide the intent classification dataset into ind and ood subset. this may not reflect real online intent classification scenarios. we design more diverse gid dataset construction strategies, mainly in order to be able to discuss the practical significance and key challenges of gid more comprehensively. we construct three kinds of benchmark datasets gid-sd (singledomain), gid-md (multiple-domain) and gid-cd (cross-domain) based on the two widely used intent datasets, clinc @xcite and banking @xcite . the three settings denote different real-world application scenarios in dialogue systems. besides, we also construct two dataset variants gid-noise and gid-imbalance to explore more severe challenges of gid tasks in real scenes. we first briefly introduce original clinc and banking datasets, then elaborate on gid dataset construction, and display the statistic of gid benchmarks. finally, we introduce evaluation metrics for the gid task, accuracy and f1 score both for ind and ood data. ## method overall architecture we extend the idea of traditional intent classification models by using pseudo ood labels. ic calculates the n -dimension crossentropy (ce) loss for labeled ind data @xcite . similarly, we can compute (n+m)-dimension ce loss both for labeled ind and unlabeled ood data where ind labels are given but ood (pseudo) labels are estimated @xcite @xcite . thus, the key challenge is to estimate ood pseudo cluster labels by transferring prior ind knowledge. we propose two kinds of frameworks, pipeline and end-to-end, shown in fig 3 . pipeline a simple idea is pipeline-based methods which firstly learn ood cluster assignments, then jointly classify labeled ind data and pseudo labeled ood data. specifically, we use the same bert intent encoder as deepaligned @xcite to cluster ood data. to transfer prior knowledge, we first pre-train the encoder on ind data to get intent representations. then, we respectively use two ood clustering methods, kmeans @xcite and deepaligned to obtain pseudo ood labels ŷood . finally, we mix up all the ind and ood data and construct the new (n+m)-dimension intent label y as follows: end-to-end the main drawback of pipeline methods is the lack of deep semantic interaction between ind and ood data in the clustering stage, leading to poor pseudo cluster labels. to alleviate the issue, we adopt an end-to-end framework to simultaneously learn pseudo ood cluster labels and classify ind&ood classes, shown in fig 3 . our motivation is that each view of an ood intent query after data augmentation can predict the other's pseudo labels, following swapped prediction @xcite . and we can learn the simple pseudo-labeling process via the unified classification loss instead of extra clustering objectives. specifically, we use the same pre-trained bert encoder in ind data as pipeline and two independent projection layers, ind head i and ood head o. given an input query, we concat the outputs of two heads as the final logit. for labeled ind intents, the ground-truth labels are easily obtained by eq 1. we now discuss how to get the pseudo labels of unlabeled ood intents. inspired by caron et al. (2020); asano et al. (2020); fini et al. (2021), we use the following swapped prediction way: deepaligned similar to k-means, the difference is that the clustering algorithm adopts deepaligned @xcite , which is the current state-ofthe-art method for ood discovery task. deepaligned-mix this is an end-to-end approach where we extend deepaligned for gid. deepaligned is an iterative clustering method. in each iteration, it firstly uses k-means and an alignment strategy to cluster and label the ood data and then computes the cross-entropy classification for representation learning. ## challenges based on the above analysis, we summarize the current challenges faced by the gid task: fine-grained ood data when ood intents are fine-grained like gid-sd, the ood performance of existing gid methods decreases significantly. we argue fine-grained ood intents make it hard to construct clear boundary while clustering. cross-domain transfer when ind and ood intent types are from different distant domains, the knowledge learned from ind is hard to transfer to ood due to the semantic gap in different domains. ood noise ood data collected in practical applications are usually noisy, and there may be some ood samples that do not belong to a certain intent type. the performance of each gid method degrades when trained with these noisy ood data. inaccurate estimation of the number of ood categories most previous work assume the number of ood categories is known. however, in practical applications, we usually need to estimate the number of categories k, which is often inaccurate. we propose a preliminary analysis in section 5.3.4 which shows significant performance drop when the estimation is not totally accurate. ood discovery aims to discover new intent concepts from unlabeled ood data and form ood intent clusters @xcite @xcite , which focuses more on how to cluster ood data, while ignoring the fusion of ind and ood, which makes the model only recognize ood intents. for example, @xcite design an iterative clustering algorithm deepaligned, which iteratively learns intent representations then cluster assignments. open intent extraction also aims to extract unknown intents from unlabelled user queries @xcite , and is a completely unsupervised task. however, in terms of method, open intent extraction is more about extracting intent names through sequence annotation methods. in contrast, gid aims to train a network that can simultaneously classify a set of labeled ind intent classes while discovering and recognizing unlabeled ood intents. incremental/continual learning there is currently some work on extending closed-set classifier to new classes in the open world incrementally, such as @xcite . but all these works follow a traditional incremental learning setting, which requires new category data with labels. in practical applications, we can only obtain these unlabeled ood data from the dialogue system logs, and these data are often updated continuously, and human annotation of these data is very labor-intensive. therefore, we propose a more human-free task gid, which aims to automatically discover new categories from the unlabeled ood data, and further expand the recognition scope of the existing ind intent classifier incrementally. zero-shot intent detection zero-shot intent detection @xcite assumes that no target domain data is available during training, but the category and category descriptions from target domain are given, but in practical applications we often have access to a large amount of unlabeled dialogue logs, and we need to consider how to discover new intent categories from them for system development. ## conclusion in this paper, we introduce a new task, generalized intent discovery (gid), which aims to extend an ind intent classifier to an open-world intent set. then we provide three public datasets for different application scenarios and establish a benchmark for the gid task. we also propose extensive baselines of two frameworks, pipeline-based and end-to-end for future work. further, we conduct exhaustive experiments and qualitative analysis to comprehend key challenges and provide new guidance for future gid research.