Enhancing the generalization for Intent Classification and Out-of-Domain Detection in SLU (2021)

Abstract: Intent classification is a major task in spoken language understanding (SLU). Since most models are built with pre-collected in-domain (IND) training utterances, their ability to detect unsupported out-of-domain (OOD) utterances has a critical effect in practical use. Recent works have shown that using extra data and labels can improve the OOD detection performance, yet it could be costly to collect such data. This paper proposes to train a model with only IND data while supporting both IND intent classification and OOD detection. Our method designs a novel domain-regularized module (DRM) to reduce the overconfident phenomenon of a vanilla classifier, achieving a better generalization in both cases. Besides, DRM can be used as a drop-in replacement for the last layer in any neural network-based intent classifier, providing a low-cost strategy for a significant improvement. The evaluation on four datasets shows that our method built on BERT and RoBERTa models achieves state-of-the-art performance against existing approaches and the strong baselines we created for the comparisons.

## introduction spoken language understanding (slu) systems play a crucial role in ubiquitous artificially intelligent voice-enabled personal assistants (pa). slu needs to process a wide variety of user utterances and carry out user's intents, a.k.a. intent classification. many deep neural network-based slu models have recently been proposed and have demonstrated significant progress @xcite @xcite @xcite in classification accuracy. these models usually apply the closed-world assumption, in which the slu model is trained with predefined domains, and the model expects to see the same data distribution during both training and testing. however, such an assumption is not held in the practical use case of pa systems, where the system is used under a dynamic and open environment with personal expressions, new vocabulary, and unknown intents that are out of the design scope. to address the challenges in open-world settings, previous works adopt varied strategies. @xcite use a cold-start algorithm to generate additional training data to cover a larger variety of utterances. this strategy relies on the software developers to pre-build all possible skills. @xcite introduce a skillbot that allows users to build up their own skills. recently, @xcite ; @xcite enables an slu model to incorporate user personalization over time. however, the above approaches do not explicitly address unsupported user utterances/intents, leading to catastrophic failures illustrated in figure 1 . thus, it is critically desirable for an slu system to classify the supported intents (in-domain (ind)) and reject unsupported ones (out-of-domain (ood)) correctly. a straightforward solution is to collect ood data and train a supervised binary classifier on both ind data and ood data @xcite . however, collecting a representative set of ood data could be impractical due to the infinite compositionality of language. arbitrarily selecting a subset could incur the selection bias, causing the learned model might not generalize to unseen ood data. @xcite avoid learning with ood data by using generative models (e.g., autoencoder and gan) to capture the ind data distribution, then judge ind/ood based on the reconstruction error or likelihood. recently, @xcite utilizes a large training data to enable the meta-learning for ood detection. @xcite generates pseudo ood data to learn the ood detector. the above-discussed approaches require additional data or training procedures beyond the intent classification task, introducing significant data collection effort or inference overhead. this paper proposes a strategy based on neural networks to use only ind utterances and their labels to learn both the intent classifier and ood detector. our strategy modifies the structure of the classifier, introducing an extra branch as a regularization target. we call the structure a domain-regularized module (drm). this structure is probabilistically motivated and empirically leads to a better generalization in both intent classification and ood detection. our analysis focuses more on the latter task, finding that drm not only outputs a class probability that is a better indicator for judging ind/ood, but also leads to a feature representation with a less distribution overlap between ind and ood data. more importantly, drm is a simple drop-in replacement of the last linear layer, making it easy to plug into any off-the-shelf pretrained models (e.g. bert @xcite ) to fine-tune for a target task. the evaluation on four datasets shows that drm can consistently improve upon previous state-of-the-art methods. ## our method our method is inspired by the decomposed confidence of generalized-odin @xcite ), but we leverage the fact that the training data are all from ind to introduce an extra regularization. this regularization leads to a better generalization (lower classification error) on the intent classification. the improvement is in contrast to the original generalized-odin, which has its classification error slightly increased. since the improved generalization is likely due to a more generalizable feature representation, we leverage this observation, providing a modified mahalanobis @xcite , which we called l-mahalanobis, for a transformerbased model to detect ood data. in the following sections, we first describe the drm and then elaborate on using the outputs of a drm-equipped model to detect ood data.