Few-Shot Intent Classification by Gauging Entailment Relationship Between Utterance and Semantic Label (2021)

Abstract: Zhang et al. (2020) proposed to formulate few-shot intent classification as natural language inference (NLI) between query utterances and examples in the training set. The method is known as discriminative nearest neighbor classification or DNNC. Inspired by this work, we propose to simplify the NLI-style classification pipeline to be the entailment prediction on the utterance-semantic-label-pair (USLP). The semantic information in the labels can thus been infused into the classification process. Compared with DNNC, our proposed method is more efficient in both training and serving since it is based upon the entailment between query utterance and labels instead of all the training examples. The DNNC method requires more than one example per intent while the USLP approach does not have such constraint. In the 1-shot experiments on the CLINC150 (Larson et al., 2019) dataset, the USLP method outperforms traditional classification approach by >20 points (in-domain ac- curacy). We also find that longer and semantically meaningful labels tend to benefit model performance, however, the benefit shrinks as more training data is available.

## introduction many methods have been considered for few-shot intent classification. a simple but often effective approach is to simply generate more data through data augmentation. @xcite @xcite explored data augmentation at token-and feature-level to boost model performance. meta-learning has also been studied extensively for few-shot learning. for instance, induction network @xcite tried to learn general class representations via episode-based meta training and predict utterance labels based on the relation score between the query utterance and classes. furthermore, large-scale pre-trained language models are often employed to mitigate the lack of annotated data for the target task. schick and schütze, 2021 leveraged pre-trained roberta @xcite and xlm-r @xcite to learn to generate task descriptions on small labeled datasets. they then use the trained models to produce descriptions and soft labels on large, task-specific unlabeled datasets and use them to train classifier. although this approach has been proven to be effective, it requires extra unlabeled data and additional human supervision on description generation task. dnnc @xcite reformulates few-shot text classification as nli-style pairwise comparison between training example and query utterance. however, dnnc requires at least two examples per intent for training and has to make m ×n (m: number of intents; n: number of training examples per intent) pairwise comparisons for each classification. along the line of nli-based classification, @xcite explored to leverage short semantic labels. however, this work is limited to zero-short setting and doesn't provide extensive analysis on how semantic information in labels affects model performance. however, the dnnc work ignores one valuable and readily available supervision in the training data, the semantics in the labels. our work is largely motivated by the hypothesis that semantic labels may carry valuable information about the intents and could benefit few-shot learning. there are prior works exploring how to leverage semantic labels in nlp tasks. for examples, @xcite has proposed to improve the prototypical network @xcite by directly embedding semantic labels; @xcite has tried to use semantic information in labels for few-shot slot tagging. to our knowledge, however, there is no known work that has explored to leverage semantic labels for nli-style intent classification. neither has any work been done to study how model per-formance changes with regard to the interplay of data augmentation, different labeling, and number of training examples. based upon dnnc, our proposed method, utterance-semantic-label-pair (uslp), also leverages nli-style classification. instead of computing the entailment relationship between query sentence and the example sentences in the support set, we use the model to gauge the entailment relationship between query text and semantic labels. the semantic information in the labels can be perfectly infused into the classification process in this way. the pairwise entailment prediction is also reduced to m times per classification compared with the dnnc's m ×n. figure 1 provides a few examples to illustrate the difference between uslp and dnnc. in the following 1-shot experiments on the clinc150 @xcite ) dataset, we show that the uslp method outperforms the standard classification approach over 20 points with respect to in-domain accuracy. it is noteworthy that the predecessor of uslp, dnnc requires more than one example per intent for training. although dnnc could do self-entailment training in 1-shot setting, our preliminary results show that the in-domain accuracy of multiple runs is extremely low (best result is below 20 from multiple runs). we also show that data augmentation, longer and more descriptive labeling, and nli pre-training could boost model performance in few shot setting. however, as more training data is available, the efficacy of these performance boosters tends to shrink or even becomes negative. our contributions can be summarized in two fold: 1, we proposed a new intent classification pipeline, uslp, and showed its effectiveness especially in 1-shot setting; 2, we studied how data augmentations, different labeling methods, and nli pre-training might impact model performance in different few shot scenarios. ## conclusion we have created a new few-shot intent classification method, uslp, based upon nli-style prediction. the uslp approach significantly outperforms traditional classification method by a large margin on 1-shot clinc150 dataset and achieves about 75% of the 100-shot traditional classifier on indomain classification with better oos performance. this outstanding result indicates that the uslp approach can be an effective solution for developers who want to quickly build an intent classifier with extremely limited amount of training data. we have also found that detailed description can further boost uslp performance, but detailed labeling also requires labelers to have deeper understanding of each intent class and thus prolongs labeling process.