Unknown Intent Detection Using Multi-Objective Optimization on Deep Learning Classifiers (2021)

Abstract: Modelling and understanding dialogues in a conversation depends on identifying the user intent from the given text. Unknown or new intent detection is a critical task, as in a realistic scenario a user intent may frequently change over time and divert even to an intent previously not encountered. This task of separating the unknown intent samples from known intents one is challenging as the unknown user intent can range from intents similar to the predefined intents to something completely different. Prior research on intent discovery often consider it as a classification task where an unknown intent can belong to a predefined set of known intent classes. In this paper we tackle the problem of detecting a completely unknown intent without any prior hints about the kind of classes belonging to unknown intents. We propose an effective post-processing method using multi-objective optimization to tune an existing neural network based intent classifier and make it capable of detecting unknown intents. We perform experiments using existing state-of-the-art intent classifiers and use our method on top of them for unknown intent detection. Our experiments across different domains and real-world datasets show that our method yields significant improvements compared with the state-of-the-art methods for unknown intent detection.

## introduction detecting whether an intent is unknown or new in a dialogue system has become an important task for improving customer satisfaction. since user intent may frequently change over time in many realistic scenarios, unknown (new) intent detection has become a crucial problem in conversational artificial intelligence (cai). this can ultimately help enhance system interaction with the customer. this task is challenging since there is no prior knowl-edge of the type or the exact numbers of unknown intents that would be encountered in the future. we model unknown intent detection as an (m+1)class classification task as suggested by @xcite @xcite and consider unknown classes as the (m+1) th class. we aim to identify the known intent samples accurately, while at the same time we focus on determining the unknown intent samples. this has to be done without any prior knowledge about the kind of unknown intents. in order to solve this problem, researchers have proposed deep neural networks like openmax @xcite , which fits weibull distribution to the outputs of the penultimate layer. another system msp @xcite calculates the softmax probability of known samples and discards the unknown samples with lower confidence. in our approach, we attempt to solve the problem of unknown intent detection with added constraints such as not having any prior knowledge of a finite set of intents. the main contributions of this paper are: the rest of the paper is organized as follows. in section 2, we present the literature survey of previous work done on this topic. in section 3 we elaborately describe the proposed methodology. in section 4, we discuss the experimental setup and the datasets used in our experiments. in section 5, we analyse the results of detecting the unknown intents. finally, section 6 concludes the paper with future work that can be explored in this field. ## related works intent detection is a much explored area in dialogue systems with a broad spectrum of literature available @xcite @xcite @xcite . most of these works are based on closed world classification that does not consider any open intent. @xcite proposed a zero-shot learning (zsl) for intent detection. however, zsl is different from our task as it only contains finite known set of classes during testing. @xcite tried to optimise the intent classifier together with an out-of-domain detector, which was trained using out-of-domain samples. the generative method proposed by @xcite used adversarial learning to generate positive and negative examples from known classes but the method did not work well in the discrete data space like text. @xcite proposed generative adversarial network (gan) to train on the id samples and use the discriminator to detect the out-of-domain samples. @xcite showed that deep generative models fail to capture the high-level semantics on real world data. @xcite fit the probability distributions to statistical extreme value theory (evt) using a weibull-calibrated multi-class support vector machine (svm) to detect the unnormalized posterior probability of inclusion for open set problems. odin @xcite enlarged the differences between known and unknown samples by using temperature scaling and input pre-processing but all the above method need negative samples for selecting the decision boundary or probability threshold. doc @xcite , instead of using softmax as the final output layer, built a multi-class classifier with a 1-vs-rest final layer which contains a sigmoid function for each seen class to reduce the open space risk. zero-shot intent classification aims to generalize knowledge and concepts learned from the seen intents to recognize unseen intents. early methods @xcite explored the relationship between seen and unseen intents by introducing external resources such as manually defined attributes or label ontologies, but they are usually expensive to obtain. to deal with this, some methods @xcite map the utterances and intent labels to an embedding space and then model their relations in the same space. intentcapsnet-zs @xcite extends capsule networks @xcite for zero-shot intent classification by transferring the prediction vectors from seen classes to unseen classes. recap-snet @xcite shows that intentcapsnet-zs hardly recognizes utterances from unseen intents in the generalized zero-shot classification scenario, and proposes to solve this issue by transferring the transformation matrices from the seen to unseen intents. these approaches also need unknown intent embedding for classifying these instances. our work do not require the assumption of that classes belong to a closed word. we don't need the unseen intent samples to get the deep learning classifier to detect unknown intents as well. ## methodology we train two different deep learning model for intent classification and use our post-processing steps on top of these to obtain optimal results. the pipeline of the system processes is shown in figure 1 . we describe the models along with our novel post-processing steps in this section. ## result and analysis we experiment with different variants of the proposed model as follows: (i). bi-lstm + normal-thresholding, (ii). bi-lstm + nsga2 and (iii). bert + normal-thresholding and (iv). bert+nsga2. we also re-implement the adm model @xcite and obtain the results on the datasets mentioned in section 4. some examples that are correctly classified by the bert+nsga2, but not by adb are shown in to "failed transfer" intent class. these close intent classes are hard to correctly predict even by humans. our model is probably learning tighter thresholds because of parallel optimization of objective functions, resulting in better performance in many cases. ## conclusion and future work in this paper, we have proposed a novel postprocessing method for unknown intent classification. after pre-training the model with labeled samples, our model can automatically learn precise thresholds to separate the known intent from unknown intent sample. our method does not require data labelled as unknown intent and can recognise open world unknown intents. our method also does not require model architecture modification of a deep learning intent classifier. extensive experiments on three benchmark datasets show that our method yields significant improvements over the compared baseline models. in future we would also like to find categories in unknown intents apart from detecting unknown intents.