Few-Shot Intent Detection via Contrastive Pre-Training and Fine-Tuning (2021)

Abstract: In this work, we focus on a more challenging few-shot intent detection scenario where many intents are fine-grained and semantically similar. We present a simple yet effective few-shot intent detection schema via contrastive pre-training and fine-tuning. Specifically, we first conduct self-supervised contrastive pre-training on collected intent datasets, which implicitly learns to discriminate semantically similar utterances without using any labels. We then perform few-shot intent detection together with supervised contrastive learning, which explicitly pulls utterances from the same intent closer and pushes utterances across different intents farther. Experimental results show that our proposed method achieves state-of-the-art performance on three challenging intent detection datasets under 5-shot and 10-shot settings.

## introduction intent detection, aiming to identify intents from user utterances, is a key component in task-oriented dialog systems. in real systems such as amazon alexa, correctly identifying user intents is crucial for downstream tasks @xcite . a practical challenge is data scarcity as it is expensive to annotate enough examples for emerging intents, and how to accurately identify intents in few-shot learning has raised attention. existing methods address the few-shot intent detection tasks mainly from two perspectives: (1) data augmentation and (2) task-adaptive training with pre-trained models. for the first category, @xcite and @xcite propose a nearest neighbor classification schema with full use of the limited training examples in both training and inference stages. @xcite and @xcite propose to generate utterances for emerging intents based on variational * work done while the first author was an intern at adobe research. however, previous methods such as data augmentation related models @xcite are inefficient for training and hard to scale to tasks with lots of intents. moreover, these models do not tackle well the following scenarios: in real scenarios, the few-shot intent detection could be more challenging when there exist many finegrained intents, especially semantically similar intents. for instance, banking77 @xcite ) has a single domain with 77 intents, and clinc150 @xcite has ten domains with 150 intents. many intents in the datasets are similar. therefore, training models is rather challenging when there are only limited examples. inspired by the recent success of contrastive learning @xcite @xcite @xcite @xcite , which aims to enhance discrimination abilities of models, this work proposes improving few-shot intent detection via contrastive pre-training and fine-tuning (cpft). intuitively, we first learn to implicitly discriminate semantically similar utterances via contrastive self-supervised pre-training on intent datasets without using any intent labels. we then jointly perform few-shot intent detection and supervised contrastive learning. the supervised contrastive learning helps the model explicitly learn to pull utterances from the same intent close and push utterances across different intents apart. our contributions are summarized as follows: ## related work since this work is related to few-shot intent detection and contrastive learning, we review recent work from both areas in this section. the few-shot intent detection task typically includes three scenarios: (1) learn a intent detection model with only k examples for each intent @xcite @xcite ; (3) learn to identify both in-domain and out-of-scope queries with only k examples for each intent @xcite @xcite . (2) given a model trained on existing intents with all examples, learn to generalize the model to new intents with only k examples for each new intent @xcite . in this work, we focus on the first scenario, and several methods have been proposed to tackle the challenge. specifically, @xcite proposes a data augmentation schema, which pretrains a model on annotated pairs from natural language inference (nli) datasets and designs the nearest neighbor classification schema to adopt the transfer learning and classify user intents. however, the training is expensive and hard to scale to tasks with hundreds of intents @xcite . @xcite propose the task-adaptive training, which leverages models pretrained from a few hundred million dialogues to tackle few-shot intent detection. it also includes an unsupervised mask language modeling loss on the target intent datasets and shows promising improvements. contrastive learning has shown superior performance on various domains, such as visual representation @xcite @xcite , graph representation @xcite , and recommender systems @xcite . moreover, recent works also adopt contrastive learning in natural language processing tasks @xcite @xcite , which employs the contrastive learning to train the encoder. specifically, @xcite designs a supervised contrastive learning loss for fine-tuning data. @xcite designs a simple contrastive learning framework through dropout and it shows state-of-the-art performance on unsupervised and full-shot supervised semantic textual similarity tasks. @xcite designs self-supervised mirror-bert framework with two types of data augmentation: randomly erase or mask parts of the input texts; feature level augmentation through dropout. our work differs from them in several respects: firstly, we specifically tackle the few-shot intent detection task rather than the general full-shot learning; secondly, we design a schema and employ contrastive learning in both self-supervised pretraining and supervised fine-tuning stages. ## cpft methodology we consider a few-shot intent detection task that handles c user intents, where the task is to classify a user utterance u into one of the c classes. we set balanced k-shot learning for each intent @xcite , i.e., each intent only includes k examples in the training data. as such, there are in total c • k training examples. in the following section, we first describe the self-supervised contrastive pre-training for utterance understanding before introducing the supervised fine-tuning for few-shot intent detection. ## ablation study and analysis is the schema with both stages necessary? we conduct ablation study to investigate the effects of self-supervised contrastive pre-training and supervised contrastive fine-tuning. is contrastive pre-training beneficial to the target intent dataset? additionally, we study whether contrastive pre-training can benefit the intent detection when excluding the target datasets. specifically, we pre-train the model on the datasets except for the hwu64 dataset on the first stage and do few-shot learning on hwu64 during the second stage. compared to the model without contrastive pre-training on the first stage, the performances are improved by 1.98% and 1.21% under 5-shot and 10-shot settings, respectively. the improvements indicate that the contrastive pre-training is helpful to transfer knowledge to new datasets. however, there are still performance drops compared to the contrastive pre-training, including the hwu64 dataset. which shows that it is beneficial to include the target dataset during self-supervised contrastive learning. we leave whether self-supervised contrastive pre-training only on the target intent dataset benefits as a future study. is the training sensitive to hyper-parameters? we also study the effects of hyper-parameters of contrastive learning, i.e., the temperature τ and weight λ . we set τ ∈ {0.05, 0.1, 0.3, 0.5} and λ ∈ {0.01, 0.03, 0.05, 0.1}. in our primary experiments, we do not find τ has a notable influence during the self-supervised contrastive pre-training on the first stage. besides, we found that a batch size larger than 32 works well in the pre-training phase. however, during the few-shot fine-tuning stage, when setting τ to a small value 0.05, which heavily enforces the penalty to hard negative examples and λ to a large value 0.1, which increases the weight of supervised contrastive learning loss, the performance drops significantly. in addition, the batch size influences performance on this stage. therefore, few-shot supervised contrastive loss is sensitive to hyper-parameters when there are limited training examples. we leave more studies to future work. ## conclusion in this paper, we improve the performance of fewshot intent detection via contrastive pre-training and fine-tuning. it first conducts self-supervised contrastive pre-training on collected intent detection datasets without using any labels, where the model implicitly learns to separate fine-grained intents. then it performs the few-shot fine-tuning based on the joint intent classification loss and supervised contrastive learning loss, where the supervised contrastive loss encourages the model to distinguish intents explicitly. experimental results on three challenging datasets show that our proposed method achieves state-of-the-art performance.