Unknown Intent Detection Using G aussian Mixture Model with an Application to Zero-shot Intent Classification (2020)

Abstract: User intent classification plays a vital role in dialogue systems. Since user intent may frequently change over time in many realistic scenarios, unknown (new) intent detection has become an essential problem, where the study has just begun. This paper proposes a semantic-enhanced Gaussian mixture model (SEG) for unknown intent detection. In particular, we model utterance embeddings with a Gaussian mixture distribution and inject dynamic class semantic information into Gaussian means, which enables learning more class-concentrated embeddings that help to facilitate downstream outlier detection. Coupled with a density-based outlier detection algorithm, SEG achieves competitive results on three real task-oriented dialogue datasets in two languages for unknown intent detection. On top of that, we propose to integrate SEG as an unknown intent identifier into existing generalized zero-shot intent classification models to improve their performance. A case study on a state-of-the-art method, ReCapsNet, shows that SEG can push the classification performance to a significantly higher level.

## introduction understanding user intent is crucial for developing conversational and dialogue systems. it is essential to accurately identify the intent behind a user utterance to better guide downstream decisions and policies. with the advent of conversational ai, dialogue systems are becoming central tools in many applications such as mobile apps, companion bots, virtual assistants and so on. since user interests may change frequently over time, the ai agents may continuously see unknown (new) user intents. manual annotation can hardly catch up with such rapid development, which motivates the problem of unknown intent detection that has recently attracted increasing interest from both academia and industry. while there have been some pioneering works studying the open-world classification problem in natural language processing @xcite , very few methods are designed for unknown intent detection. to our knowledge, the first work is by @xcite , in which the authors use large margin cosine loss (lmcl) to learn deep discriminative features and then feed them to a density-based outlier detection algorithm to identify unknown intents. although this method performs well on some benchmark datasets, it has two limitations. (1) in training, lmcl ignores the prior knowledge of class labels, while it has been shown that label correlations captured in the embedding space can improve prediction performance, especially in the zero-shot learning scenarios @xcite . (2) lmcl computes the cosine distance between embeddings in the feature space and trains with a softmax cross-entropy loss, making the embedding distribution of each class long and narrow @xcite , which may be less suitable for applying density-based anomaly detection algorithms to detect unknown intents. in this paper, we aim to address these limitations and propose a novel semantic-enhanced gaussian mixture model (seg) for unknown intent detection. in contrast to the softmax function, the gaussian mixture model enforces embeddings to form ball-like dense clusters in the feature space, which may be more desirable for outlier detection, especially when using density-based outlier detection algorithms. furthermore, we propose to inject the semantic information of class labels into the gaussian mixture distribution by assigning the embeddings of class labels or descriptions to be the means of the gaussians. this enables seg to learn more class-concentrated embeddings that can benefit downstream outlier detection. we further use a large margin loss to make seg learn more discriminative features and employ a density-based outlier detection algorithm lof @xcite to detect unknown intents. identifying unknown intents is not enough for some application scenarios where it is important to know what exactly the new intents are, e.g., zero-shot intent classification. current generalized zero-shot intent classification methods @xcite @xcite attempt to classify test instances directly by making predictions in the pool of all the seen and unseen intents. however, their prediction performances are quite low, and they are still far from practical use. in this work, we propose to integrate seg as an unknown intent identifier into the generalized zero-shot intent classification pipeline. the basic idea is that correctly identifying if the intent of an utterance is known or unknown will make the subsequent intent classification task much easier. we conduct a case study on a state-of-the-art zero-shot intent classification method recapsnet @xcite . the results show that incorporating seg successfully improves the performance of recapsnet by a large margin. it even pushes the performance to a practical level on the snips dataset @xcite . the main contributions of this paper are summarized as follows. the rest of the paper is organized as follows. in section 2, we review related works on intent classification and open-world classification. in section 3, we discuss the proposed seg model in details. in section 4, we present experimental results on unknown intent detection. in section 5, we apply seg to improve generalized zero-shot intent classification and conduct a case study. finally, section 6 concludes the paper. ## experiments in this section, we present experimental results on unknown intent detection. formally, we train an ## application in generalized zero-shot intent classification in this section, we apply our method seg to an extended application of unknown intent classification -zero-shot intent classification. it aims to discriminate unseen intents, which is beyond only detecting their existence. specifically, given the training data d @xmath0, y tr ) where y @xmath1, a zero-shot classification system is trained to predict the label ŷte of any test sample which may belong to an unseen class, using the knowledge transferred from the seen data. there are two common settings for zero-shot learning, generalized zero-shot classification, where ŷte ∈ {c seen , c unseen }, and standard zero-shot classification, where ŷte ∈ c unseen . here, c unseen is the set of unseen intent classes. previous attempts try to tackle the challenge of zero-shot intent classification from three directions. in this work, we improve generalized zero-shot intent classification by integrating the proposed seg model as a binary unknown intent identifier into the original pipeline. we explore multiple ways of integration and conduct a case study based on a state-of-the-art method recapsnet @xcite . ## conclusion in this paper, we have proposed seg, a semanticenhanced gaussian mixture model coupled with a lof outlier detector, for unknown (new) intent detection. we empirically verified the effectiveness of seg for unknown intent detection on real dialogue datasets in english and chinese. furthermore, we successfully applied seg to improve generalized zero-shot intent classification and achieved remarkable performance gain over a most recent competitive method recapsnet. in future work, we plan to conduct more empirical studies on seg and further improve its performance on new intent identification. we also plan to conduct more case studies in applying seg to boost the performance of current zero-shot intent classification methods.