Estimating Soft Labels for Out-of-Domain Intent Detection (2022)

Abstract: Out-of-Domain (OOD) intent detection is important for practical dialog systems. To alleviate the issue of lacking OOD training samples, some works propose synthesizing pseudo OOD samples and directly assigning one-hot OOD labels to these pseudo samples. However, these one-hot labels introduce noises to the training process because some “hard” pseudo OOD samples may coincide with In-Domain (IND) intents. In this paper, we propose an adaptive soft pseudo labeling (ASoul) method that can estimate soft labels for pseudo OOD samples when training OOD detectors. Semantic connections between pseudo OOD samples and IND intents are captured using an embedding graph. A co-training framework is further introduced to produce resulting soft labels following the smoothness assumption, i.e., close samples are likely to have similar labels. Extensive experiments on three benchmark datasets show that ASoul consistently improves the OOD detection performance and outperforms various competitive baselines.

## introduction intent detection is essential for dialogue systems, and current methods usually achieve high performance under the closed-world assumption @xcite , i.e., data distributions are static, and only a fixed set of intents are considered. however, such an assumption may not be valid in practice, where we usually confront an open-world @xcite , i.e., unknown intents that are not trained may emerge. it is necessary to equip dialogue systems with out-of-domain (ood) detection abilities so that they can accurately classify known in-domain (ind) intents while rejecting unknown ood intents @xcite . a major challenge for ood detection is the lack of ood samples @xcite . in most applications, it is hard, if not impossible, to collect ood samples from the test distribution before training @xcite . to tackle this issue, various studies try to synthesize pseudo ood samples in the training process. existing methods include distorting ind samples @xcite @xcite , using generative models @xcite , or even mixing-up ind features @xcite . promising results are reported by training a (k + 1)-way classifier (k ind classes + 1 ood class) using these pseudo ood samples @xcite . this classifier can classify ind intents while detecting ood intent since inputs that fall into the ood class are regarded as ood inputs. previous studies directly assign one-hot ood labels to pseudo ood samples when training the (k + 1)-way classifier @xcite . however, this scheme brings noise to the training process because "hard" pseudo ood samples, i.e., ood samples that are close to ind distributions, may carry ind intents @xcite (see figure 1 ). indiscriminately assigning one-hot ood labels ignores the semantic connections between pseudo ood samples and ind intents. moreover, this issue becomes more severe as most recent studies are dedicated to producing hard pseudo ood samples @xcite since these samples are reported to facilitate ood detectors better @xcite . collisions between pseudo ood samples and ind intents will be more common. we argue that ideal labels for pseudo ood samples should be soft labels that allocate non-zero probabilities to all intents @xcite . specifically, we demonstrate in §3.2 that pseudo ood samples generated by most existing approaches should be viewed as unlabeled data since they may carry both ind and ood intents. soft labels help capture the semantic connections between pseudo ood samples and ind intents. moreover, using soft labels also conforms to the smoothness assumption, i.e., samples close to each other are likely to receive similar labels. this assumption lays a foundation for modeling unlabeled data in various previous works @xcite . in this study, we propose an adaptive soft pseudo labeling (asoul) method that can estimate soft labels for given pseudo ood samples and thus help to build better ood detectors. specifically, we first construct an embedding graph using supervised contrastive learning to capture semantic connections between pseudo ood samples and ind intents. following the smoothness assumption, a graph-smoothed label is produced for each pseudo ood sample by aggregating nearby nodes on the graph. a co-training framework with two separate classification heads is introduced to refine these graph-smoothed labels. concretely, the prediction of one head is interpolated with the graphsmoothed label to produce the soft label used to enhance its peer head. the final ood detector is formulated as a (k + 1)-way classifier with adaptive decision boundaries. extensive experiments on three benchmark datasets demonstrate that asoul can be used with a wide range of ood sample generation approaches and consistently improves the ood detection performance. asoul also helps achieve new state-ofthe-art (sota) results on benchmark datasets. our major contributions are summarized: ## conclusion in this paper, we first analyze the limitation of existing ood detection approaches that use one-hot labeled pseudo ood samples. then we propose a method asoul that can estimate soft labels for given pseudo ood samples and use these soft labels to train better ood detectors. an embedding space is constructed to produce graph-smoothed labels to capture the semantic connections between ood samples and ind intents. a co-training framework further refines these graph-smoothed labels. experiments demonstrate that our method can be combined with different pseudo ood sample generation approaches, and it helps achieve sota results on three benchmark datasets. in the future, we plan to apply our method in other tasks, such as text-to-sql parsers @xcite @xcite or lifelong learning @xcite .