Discriminative Nearest Neighbor Few-Shot Intent Detection by Transferring Natural Language Inference (2020)

Abstract: Intent detection is one of the core components of goal-oriented dialog systems, and detecting out-of-scope (OOS) intents is also a practically important skill. Few-shot learning is attracting much attention to mitigate data scarcity, but OOS detection becomes even more challenging. In this paper, we present a simple yet effective approach, discriminative nearest neighbor classification with deep self-attention. Unlike softmax classifiers, we leverage BERT-style pairwise encoding to train a binary classifier that estimates the best matched training example for a user input. We propose to boost the discriminative ability by transferring a natural language inference (NLI) model. Our extensive experiments on a large-scale multi-domain intent detection task show that our method achieves more stable and accurate in-domain and OOS detection accuracy than RoBERTa-based classifiers and embedding-based nearest neighbor approaches. More notably, the NLI transfer enables our 10-shot model to perform competitively with 50-shot or even full-shot classifiers, while we can keep the inference time constant by leveraging a faster embedding retrieval model.

## introduction intent detection is one of the core components when building goal-oriented dialog systems. the goal is to achieve high intent classification accuracy, and another important skill is to accurately detect unconstrained user intents that are out-ofscope (oos) in a system @xcite . a practical challenge is data scarcity because different systems define different sets of intents, and thus few-shot learning is attracting much attention. however, previous work has mainly focused on the few-shot intent classification without oos @xcite . oos detection can be considered as out-ofdistribution detection @xcite . recent work has shown that large-scale pre-trained models like bert @xcite and roberta @xcite still struggle with out-of-distribution detection, despite their strong in-domain performance @xcite . figure 1 (a) shows how unseen input text is mapped into a feature space, by a roberta-based model for 15-way 5-shot intent classification. the separation between oos and some in-domain intents is not clear, which presumably hinders the model's oos detection ability. this observation calls for investigation into more sample-efficient approaches to handling the in-domain and oos examples accurately. in this paper, we tackle the task from a different angle, and propose a discriminative nearest neighbor classification (dnnc) model. instead of expecting the text encoders to be generalized enough to discriminate both the in-domain and oos examples, we make full use of the limited training examples both in training and inference time as a nearest neighbor classification schema. we leverage the bert-style paired text encoding with deep self-attention to directly model relations between pairs of user utterances. we then train a matching model as a pairwise binary classifier to estimate whether an input utterance belongs to the same class of a paired example. we expect this to free the model from having the oos separation issue in figure 1 (a) by avoiding explicit modeling of the intent classes. unlike an embedding-based matching function as in relation networks @xcite (figure 1 (b) ), the deep pairwise matching function produces clear separation between the indomain and oos examples (figure 1 (c) ). we further propose to seamlessly transfer a natural language inference (nli) model to enhance this clear separation (figure 1 (d) ). we verify our hypothesis by conducting extensive experiments on a large-scale multi-domain intent detection task with oos @xcite in various few-shot learning settings. our experimental results show that, compared with roberta classifiers and embedding nearest neighbor approaches, our dnnc attains more stable and accurate performance both in in-domain and oos accuracy. moreover, our 10-shot model can perform competitively with a 50-shot or even full-shot classifier, with the performance boost by the nli transfer. we also show how to speedup our dnnc's inference time without sacrificing accuracy. ## proposed method this section first describes how to directly model inter-utterance relations in our nearest neighbor classification scenario. we then introduce a binary classification strategy by synthesizing pairwise examples, and propose a seamless transfer of nli. finally, we describe how to speedup our method's inference process. ## experimental results this section shows our experimental results. appendix c shows some additional figures. ## discussions and related work interpretability interpretability is an important line of research recently @xcite @xcite . the nearest neighbor approach @xcite is appealing in that we can explicitly know which training example triggers each prediction. call for better embeddings emb-knn and rn-knn are not as competitive as dnnc. this encourages future work on the task-oriented evaluation of text embeddings in knn. training time our dnnc method needs longer training time than that of the classifier (e.g., 90 vs. 40 seconds to train a single-domain model), because we synthesize the pairwise examples. as a first step, we used all the training examples to investigate the effectiveness, but it is an interesting direction to seek more efficient pairwise training. distilled model another way to speedup our model is to use distilled pre-trained models @xcite . we replaced the roberta model with a distilled roberta model, and observed large variances with significantly lower oos accuracy. @xcite also suggested that the distilled models would not be robust to out-of-distribution examples. few-shot text classification few-shot classification @xcite has been applied to text classification tasks @xcite @xcite , and few-shot intent detection is also studied but without oos @xcite . there are two common scenarios: 1) learning with plenty of examples and then generalizing to unseen classes with a few examples, and 2) learning with a few examples for all seen classes. meta-learning @xcite is widely studied in the first scenario. in our paper, we have focused on the second scenario, assuming that there are only a limited number of training examples for each class. our work is related to metric-based approaches such as matching networks @xcite , prototypical networks @xcite and relation networks @xcite , as they model nearest neighbours in an example-embedding or a classembedding space. we showed that a relation network with the roberta embeddings does not perform comparably to our method. we also considered several ideas from prototypical networks @xcite , but those did not outperform our emb-knn baseline. these results indicate that deep self-attention is the key to the nearest neighbor approach with oos detection. ## conclusion in this paper, we have presented a simple yet efficient nearest-neighbor classification model to detect user intents and oos intents. it includes paired encoding and discriminative training to model relations between the input and example utterances. moreover, a seamless transfer from nli and a joint approach with fast retrieval are designed to improve the performance in terms of the accuracy and inference speed. experimental results show superior performance of our method on a large-scale multi-domain intent detection dataset with oos. future work includes its cross-lingual transfer and cross-dataset (or cross-task) generalization. dataset snli wnli mnli size of the development set 9999 70 9814 accuracy 94.5% 41.4% 92.1% pre-training on nli tasks for the pre-training on nli tasks, we fine-tune a roberta-base model on three publicly available datasets, i.e., snli @xcite , mnli @xcite , and wnli @xcite from the glue benchmark @xcite . the optimizer and gradient clipping follow the above configurations. the number of training epochs is set to 4; the batch size is set to 32; the learning rate is set to 2e -5. we use a linear warmup learning-rate scheduling with a proportion of 0.06 by following @xcite . the evaluation results on the development sets are shown in text pre-processing for all the roberta-based models, we used the roberta roberta-base's tokenizer provided in the transformers library. 13 we did not perform any additional pre-processing in our experiments. hyper-parameter settings we then select the best hyper-parameter set along with the corresponding threshold, which achieves the best j in oos in equation ( 9 ) on the development set, among all the possible hyper-parameter sets. finally, we apply the selected model and the threshold to the test set. we follow the same process for the all-domain experiments, except that we run each experiment five times.