Effectiveness of Pre-training for Few-shot Intent Classification (2021)

Abstract: This paper investigates the effectiveness of pre-training for few-shot intent classification. While existing paradigms commonly further pre-train language models such as BERT on a vast amount of unlabeled corpus, we find it highly effective and efficient to simply fine-tune BERT with a small set of labeled utterances from public datasets. Specifically, fine-tuning BERT with roughly 1,000 labeled data yields a pre-trained model – IntentBERT, which can easily surpass the performance of existing pre-trained models for few-shot intent classification on novel domains with very different semantics. The high effectiveness of IntentBERT confirms the feasibility and practicality of few-shot intent detection, and its high generalization ability across different domains suggests that intent classification tasks may share a similar underlying structure, which can be efficiently learned from a small set of labeled data. The source code can be found at https://github.com/hdzhang-code/IntentBERT .

## introduction task-oriented dialogue systems have been widely deployed to a variety of sectors @xcite @xcite , ranging from shopping @xcite to medical services @xcite , to provide interactive experience. training an accurate intent classifier is vital for the development of such task-oriented dialogue systems. however, an important issue is how to achieve this when only limited number of labeled instances are available, which is often the case at the early development stage. to tackle few-shot intent detection, some recent attempts employ induction network @xcite , generation-based methods @xcite 2020a,b), metric learning @xcite , or self-training @xcite . these works mainly focus on designing novel algorithms for representation learning and inference, which often comes with complicated models. most recently, large-scale pre-trained language models such as bert @xcite @xcite have shown great promise in many natural language understanding tasks @xcite , and there has been a surge of interest in fine-tuning the pre-trained language models for intent detection @xcite @xcite @xcite . while fine-tuning pre-trained language models on large-scale annotated datasets has yielded significant improvements in many tasks including intent detection, it is laborious and expensive to construct large-scale annotated datasets in new application domains. therefore, recent efforts have been dedicated to adapting pre-trained language models to a specific task such as intent detection by conducting continued pre-training @xcite on a large unlabeled dialogue corpus with a specially designed optimization objective. below we summarize the most related works in this line of research for few-shot intent detection. while these methods have achieved impressive performance, they heavily rely on the existence of a large-scale corpus @xcite that is close in semantics to the target domain or consists of similar tasks for continued pre-training, which needs huge effort for data collection and comes at a high computational cost. more importantly, they completely ignore the "free lunch" -the publicly available, high-quality, manually-annotated intent detection benchmarks. for example, the dataset oos @xcite provides labeled utterances across 10 different domains. hence, our study in this paper centers around the following research question: in this paper, we provide an affirmative answer to this question. we fine-tune bert using a simple standard supervised training with approximately 1,000 labeled utterances from public datasets and obtain a pre-trained model, called intentbert. it can be directly applied for few-shot intent classification on a target domain that is drastically different from the pre-training data and significantly outperform existing pre-trained models, without further fine-tuning on target data (labeled or unlabeled). this simple "free-lunch" solution not only confirms the feasibility and practicality of few-shot intent detection, but also provides a ready-to-use well-performing model for practical use, saving the effort in algorithm design and data collection. moreover, the high generalization ability of in-tentbert on cross-domain few-shot classification tasks, which are generally considered very difficult due to large domain gaps and the few data constraint, suggests that most intent detection tasks probably share a common underlying structure that could be learned from a small set of data. further, to leverage unlabeled data in the target domain, we design a joint pre-training scheme, which simultaneously optimizes the classification error on the source labeled data and the language modeling loss on the target unlabeled data. this joint-training scheme can learn better semantic representations and significantly outperforms existing two-stage pre-training methods @xcite . a visualization of the embedding spaces produced by strong baselines and our methods is provided in fig. 1 , which clearly demonstrates the superiority of our pre-trained models. ## methodology we present a continued pre-training framework for intent classification based on the pre-trained language model bert @xcite . our pre-training method relies on the existence of a small labeled dataset d labeled @xmath0, y i )}, where y i is the label of utterance x i . such data samples can be readily obtained from public intent detection datasets such as oos @xcite and hwu64 @xcite . as will be shown in the experiments, roughly 1, 000 examples from either oos or hwu64 are enough for the pretrained intent detection model to achieve a superior performance on drastically different target domains such as "covid-19". we further consider a scenario that unlabeled utterances d unlabeled @xmath1, and propose a joint pre-training scheme that is empirically proven to be highly effective. ## conclusion we have proposed intentbert, a pre-trained model for few-shot intent classification, which is obtained by fine-tuning bert on a small set of publicly available labeled utterances. we have shown that using small task-relevant data for fine-tuning is far more effective and efficient than current practice that fine-tunes on a large labeled or unlabeled dialogue corpus. this finding may have a wide implication for other tasks besides intent detection.