Towards Multi-label Unknown Intent Detection (2022)

Abstract: Multi-class unknown intent detection has made remarkable progress recently. However, it has a strong assumption that each utterance has only one intent, which does not conform to reality because utterances often have multiple intents. In this paper, we propose a more desirable task, multi-label unknown intent detection, to detect whether the utterance contains the unknown intent, in which each utterance may contain multiple intents. In this task, the unique utterances simultaneously containing known and unknown intents make existing multi-class methods easy to fail. To address this issue, we propose an intuitive and effective method to recognize whether All Intents contained in the utterance are Known (AIK). Our high-level idea is to predict the utterance’s intent number, then check whether the utterance contains the same number of known intents. If the number of known intents is less than the number of intents, it implies that the utterance also contains unknown intents. We benchmark AIK over existing methods, and empirical results suggest that our method obtains state-of-the-art performances. For example, on the MultiWOZ 2.3 dataset, AIK significantly reduces the FPR95 by 12.25% compared to the best baseline.

## introduction intent classification is a crucial component of taskoriented dialogue systems, which aims to map the utterance to the known intent set. in an open environment, it is nearly impossible that dialogue systems are only exposed to utterances with known intents, i.e., in-distribution (ind) utterances. therefore, unknown intent detection is proposed to identify the out-of-distribution (ood) utterance, which contains the unknown intent @xcite . it can prevent dialogue systems from generating unrelated responses to ensure good user utterance i am looking to stay at the lovell lodge hotel and to see the areas local attractions. intent inform-hotel-name, request-attraction-area recent works follow the assumption that each utterance has only one intent and focus on multi-class unknown intent detection @xcite @xcite . based on this assumption, a rather popular strategy to perform ood detection relies on the maximum classifier output. for example, @xcite propose using the maximum binary classifier output. an utterance will be regarded as containing the known intent and classified as ind if the output is larger than the predefined threshold, otherwise it will be classified as ood. nevertheless, the above assumption is too strong: several intents are usually expressed in an utterance in a real-world scenario. for example, @xcite show that 52% of utterances include multi-label intents in the amazon internal dataset. it is obvious that the multi-class unknown intent detection research cannot fully meet the needs of dialogue systems. in this work, we propose a more practical task, multi-label unknown intent detection, which is to detect whether the user utterance contains unknown intents, where each utterance may contain multiple intents. we summarize three types of utterances for unknown intent detection in the multi-label setting: 1) ind utterances, only containing known intents; 2) pure ood utterances, only containing unknown intents; and 3) mixed ood utterances simultaneously containing known and unknown in-tents (see the existence of mixed ood utterances brings a great challenge for multi-label unknown intent detection, which makes the existing strategy easy to fail. as shown in figure 1 , such methods will regard the mixed ood utterances as ind utterances once detecting the known intents. to address the above issue, we propose a novel method, by recognizing whether all intents of the utterance are known (aik), to detect both pure and mixed ood utterances for multi-label unknown intent detection. overall, we first predict the number of intents contained in the utterance. then we check whether the utterance contains the same number of known intents by measuring the probability density of the utterance's known intent-wise representations. specifically, we assume the known intent-wise representation can be fitted well by a conditional gaussian distribution, then we can estimate its probability density via the mahalanobis distance. we empirically demonstrate that aik can significantly improve ood detection performance, especially for mixed ood utterances. to summarize, the key contributions of the paper are as follows: ## task formulation multi-label unknown intent detection breaks the assumption that each utterance only contains one intent, allowing each utterance contain multiple intents. it aims to detect ood utterances that contain unknown intents. formally, given a training dataset @xmath0, ## approach to perform multi-label unknown intent detection, we propose a novel method aik. in this section, we first introduce the overall idea of aik, then present its model architecture and training objective. ## conclusion in this paper, we propose a valuable and practical research task multi-label unknown intent detection. it aims to detect ood utterances that may contain multiple intents. we further propose a novel aik method to perform multi-label unknown intent detection, by recognizing whether all intents contained in the utterance are known. experimental results on two datasets validate the effectiveness of our method. we also analyze the challenges of detecting mixed ood utterances for multi-label unknown intent detection through experiments.