Disentangled Knowledge Transfer for OOD Intent Discovery with Unified Contrastive Learning (2022)

Abstract: Discovering Out-of-Domain(OOD) intents is essential for developing new skills in a task-oriented dialogue system. The key challenge is how to transfer prior IND knowledge to OOD clustering. Different from existing work based on shared intent representation, we propose a novel disentangled knowledge transfer method via a unified multi-head contrastive learning framework. We aim to bridge the gap between IND pre-training and OOD clustering. Experiments and analysis on two benchmark datasets show the effectiveness of our method.

## introduction out-of-domain (ood) intent discovery aims to group new unknown intents into different clusters, which helps improve the dialogue system for future development. compared to existing text clustering tasks, ood discovery considers how to leverage the prior knowledge of known in-domain (ind) intents to enhance discovering unknown ood intents, which makes it challenging to directly apply existing clustering algorithms @xcite @xcite to the ood discovery task. previous unsupervised ood discovery models @xcite @xcite only model ood data but ignore prior knowledge of in-domain data thus suffer from poor performance. therefore, recent work @xcite focus more on the semi-supervised setting where they firstly pre-train an in-domain intent classifier then perform clustering algorithms on extracted ood intent representations by the pre-trained ind intent classifier. for example, lin et al. (2020) firstly pre-trains a bert-based @xcite ind intent classifier then uses intent representations to perform a pairwise clustering algorithm @xcite . further, @xcite proposes an iterative clustering method, deepaligned, to obtain pseudo supervised signals using k-means @xcite . however, all of these methods ignore the matching between ind pre-training stage and ood clustering stage because they formulate ind pre-training as the classification task while ood clustering as the text clustering task. the different learning objectives make it hard to transfer prior ind knowledge to ood. besides, previous work only transfer a single intent representation from the pre-trained ind classifier to ood clustering. considering the entanglement of the intent representation, simply transferring ind features may harm ood clustering. for example, there exist two levels of intent features, instancelevel and class-level knowledge in the pre-trained ind classifier. decoupling different levels of intent features helps better knowledge transferability. to solve the issues, we propose a novel disentangled knowledge transfer method (dkt) via a unified multi-head contrastive learning framework to transfer disentangled ind intent representations to ood clustering. the main intuition is how to perform better knowledge transfer. as shown in fig 1 , we decouple the pre-trained intent representations into two independent subspaces, instance-level and class(cluster)-level using a uni-fied contrastive learning framework. different from existing ood discovery work, we equip the traditional ind pre-training stage with a similar contrastive objective as the clustering stage. specifically, we firstly learn intent features using a context encoder like bert, then add two independent transformation heads (instance-level head f and class-level head g) on top of bert. in the ind pre-training stage, we use the head f to perform supervised instance-level contrastive learning @xcite @xcite and the head g to compute traditional classification loss like cross-entropy. in the ood clustering stage, we employ similar objectives for these two heads where f is still used for instance-level contrastive learning and g is used to perform class(cluster)-level contrastive learning @xcite . we leave the details in the following section 2. using the unified contrastive objectives for pre-training and clustering bridges the gap between the two stages. besides, the two independent heads decouple the instance-and cluster-level contrastive learning to learn disentangled intent representations for better knowledge transfer. section 4 demonstrates the effectiveness of multi-head disentanglement. our contributions are three-fold: then we decouple the intent representations into two independent subspaces and use a unified contrastive learning framework to perform both ind pre-training and ood clustering. ## conclusion in this paper, we propose a novel disentangled knowledge transfer method (dkt) via a unified multi-head contrastive learning framework to transfer disentangled ind intent representations to ood clustering. experiments and analysis on two benchmarks demonstrate the effectiveness of dkt for ood discovery. we hope to explore more selfsupervised representation learning methods for ood discovery in the future.