Revisit Few-shot Intent Classification with PLM s: Direct Fine-tuning vs. Continual Pre-training (2023)

Abstract: We consider the task of few-shot intent detection, which involves training a deep learning model to classify utterances based on their underlying intents using only a small amount of labeled data. The current approach to address this problem is through continual pre-training, i.e., fine-tuning pre-trained language models (PLMs) on external resources (e.g., conversational corpora, public intent detection datasets, or natural language understanding datasets) before using them as utterance encoders for training an intent classifier. In this paper, we show that continual pre-training may not be essential, since the overfitting problem of PLMs on this task may not be as serious as expected. Specifically, we find that directly fine-tuning PLMs on only a handful of labeled examples already yields decent results compared to methods that employ continual pre-training, and the performance gap diminishes rapidly as the number of labeled data increases. To maximize the utilization of the limited available data, we propose a context augmentation method and leverage sequential self-distillation to boost performance. Comprehensive experiments on real-world benchmarks show that given only two or more labeled samples per class, direct fine-tuning outperforms many strong baselines that utilize external data sources for continual pre-training. The code can be found at https://github.com/hdzhang-code/DFTPlus .

## introduction intent detection is a critical module in task-oriented dialogue systems. the target is to classify utterances according to user intents. recent progress in intent detection highly relies on deep models and datasets with well-crafted annotations. using largescale models or datasets has been recognized as a de facto recipe for many tasks in natural language * corresponding author. processing (nlp) including intent detection . however, large training datasets are often not available due to the cost of labeling. therefore, few-shot intent detection, which aims to train a classifier with only a few labeled examples, has attracted considerable attention in recent years @xcite @xcite . the main obstacle for few-shot learning is commonly believed to be overfitting, i.e. the model trained with only a few examples tends to overfit to the training data and perform much worse on test data @xcite . to alleviate the problem, the mainstream approach is to transfer knowledge from external resources such as another labeled dataset, which has been widely used for few-shot image classification @xcite and few-shot intent detection @xcite @xcite . since recently emerged large-scale pre-trained language models (plms) have achieved great success in various nlp tasks, most recent few-shot intent detection methods propose to fine-tune plms on external resources before applying them on the target task, which is known as continual pretraining @xcite , as illustrated in fig 1 . the external resources utilized for continual pre-training include conversational corpus @xcite @xcite , natural language understanding datasets @xcite , public intent detection datasets @xcite , and paraphrase corpus @xcite . while these methods have achieved state-of-theart results, the use of external training corpora induces extra data processing effort (e.g., @xcite uses 83 million sentence pairs from 12 datasets) as well as model bias (e.g., the trained model may be biased to the intent classes used in continual pre-training) @xcite @xcite . it is commonly believed that directly fine-tuning plms with a small amount of data may generate unacceptable variance @xcite . however, it has been recently found that the instability may be caused by incorrect use of optimizer and insufficient training @xcite . further, some studies @xcite have revealed that in sentiment analysis and paraphase detection tasks, when directly fine-tuned with a small dataset, plms sush as bert @xcite demonstrate a certain level of resilience to overfitting. therefore, a thorough investigation is needed to explore the direct fine-tuning of plms for few-shot intent detection. in this work, we make the following contributions: ## direct fine-tuning we investigate a straightforward approach for fewshot intent detection -directly fine-tuning (dft) plms with the few-shot data at hand. however, it is a common belief that such a process may lead to severe overfitting. before going into detail, we first formally define the problem. ## push the limit of direct fine-tuning to push the limit of few-shot intent detection with only a few labeled data at hand and without using any external training corpora, dft++ introduces two mechanisms, as shown in fig. 2 . the first is a novel context augmentation mechanism, wherein the few data are used to prompt a generative plm to generate contextually relevant unlabeled utterances to better model target data distribution. the second is a sequential self-distillation mechanism. ## related works few-shot intent detection before the era of plms, the study of few-shot intent detection focuses on model architecture @xcite @xcite . recently, finetuning plms has become the mainstream methodology. @xcite fine-tune pair-wise encoder on natural language inference (nli) tasks. zhang et al. (2021b) fine-tune plms in a contrastive manner. zhang et al. (2021a) leverage public intent detection dataset, which is further improved by isotropization @xcite . other settings are also studied, including semi-supervised learning @xcite and incremental learning @xcite . unlike the mainstream strategy, our method does not require continual pre-training on extra resources. continual pre-training of plms continual pretraining of plms is helpful @xcite @xcite . for dialogue understanding, many works leverage conver-sational corpus to perform continual pre-training. li et al. (2020) conducts continual pre-training with a dialogue-adaptive pre-training objective and a synthesized in-domain corpus. wu et al. (2020b) further pre-trains bert with dialogue corpora through masked language modeling and contrastive loss. henderson et al. (2020) use reddit conversational corpus to pre-train a dual-encoder model. vulić et al. (2021) adopts adaptive conversational fine-tuning on a dialogue corpus. plm-based data augmentation rosenbaum et al. (2022) fine-tune plms to generate data for intent detection and slot tagging. jolly et al. (2020) develop novel sampling strategies to improve the generated utterances. kumar et al. (2022) pre-train a token insertion plm for utterances generation. however, these methods require slot values, which are assumed unavailable in this work. papangelis et al. (2021) fine-tune plms with reinforcement learning, but our augmentation method adopts offthe-shelf plm without further training. the closest work to ours is sahu et al. (2022), which utilizes off-the-shelf plms for data augmentation. however, our method focuses solely on leveraging contextual relevance to achieve improved robustness. plm-based data augmentation has been explored for other tasks, e.g. sentiment classification (yoo @xcite @xcite and natural language inference @xcite . however, these approaches may fail to scale to intent detection tasks with tens of intent classes, as shown by @xcite and our experiments. ## conclusions and limitations we revisit few-shot intent detection with plms by comparing two approaches: direct fine-tuning and continual pre-training. we show that the overfitting issue may not be as significant as commonly believed. in most cases, our proposed framework, dft++, demonstrates superior performance compared to mainstream continual pre-training methods that rely on external training corpora. one limitation of dft++ is the computational overhead caused by generative plms. additionally, our current approach includes all utterances generated by the plm, even those that might lack contextual relevance or contain noise. these issues are left for future exploration. observed that dft++ outperforms many competitive methods fine-tuned on extra data even when the number of labeled data is small. keywords used to collect the corpus for an alternative context augmentation method as introduced in subsection 3.4, one alternative context augmentation method involves manually collecting a domain-specific corpus. we experiment with banking77. to collect an external corpus, we extract web pages from wikipedia foot_4 with keywords closely related to "banking", such as "bank" and "credit card". the adopted keywords are summarized in