Exploring Zero and Few-shot Techniques for Intent Classification (2023)

Abstract: Conversational NLU providers often need to scale to thousands of intent-classification models where new customers often face the cold-start problem. Scaling to so many customers puts a constraint on storage space as well. In this paper, we explore four different zero and few-shot intent classification approaches with this low-resource constraint: 1) domain adaptation, 2) data augmentation, 3) zero-shot intent classification using descriptions large language models (LLMs), and 4) parameter-efficient fine-tuning of instruction-finetuned language models. Our results show that all these approaches are effective to different degrees in low-resource settings. Parameter-efficient fine-tuning using T-few recipe on Flan-T5 yields the best performance even with just one sample per intent. We also show that the zero-shot method of prompting LLMs using intent descriptions is also very competitive.

## introduction intent classification is the primary natural language understanding task for a virtual agent or a chatbot. providing intent-utterances for training intent classification models is a laborious process. in this paper, we address this problem by exploring zero and few-shot intent identification using large language models (llms) as well as instruction finetuned models. zero-shot and few-shot intent prediction completely remove or substantially reduce the work to provide intent-utterances, respectively. we demonstrate that the following four approaches work well in practice for zero/few-shot intent classification. here is the outline of the rest of the paper. in section 2 we describe the related work. in section 3 we detail the datasets used. in section 4 we describe the four approaches covered in this work for zero/few-shot intent classification. finally, we conclude with observations in sections 5 and 6. ## related work recent work has successfully used domain adaptation and contrastive learning for few-shot intent classification. one approach is to use embeddings from a bert model @xcite pretrained on domain data to search for utterances belonging to new intents in the domain @xcite . in a similar vein, @xcite finetune a bert model on few-shot data using contrastive learning which learns to discriminate between semantically similar sentences. our work on domain adaptation differs from these mainly due to our setting which involves serving thousands of customers. for legal reasons, we cannot co-mingle data from these customers to pre-train a single model. instead, we pre-train a sentence encoder based on an intent taxonomy and out-of-the-box intents, which consist of human generated synthetic data. in this setting, we can only train very lightweight models for each customer, e.g. a dense layer on top of a pre-trained sentence encoder. data augmentation is another widely used technique to solve the problem of data scarcity. recent work on data augmentation has focused on using multiple methods to improve model performance @xcite . llms like gpt-3 @xcite ) can be prompted to generate labeled training data for intent classification @xcite . the quality of generated training data using llms is highly dependent on the prompts. in this work, we show various prompt-based approaches that generate diverse data for training and boost the performance of intent classifiers. as the usage of conversational agents grows, it is important for them to generalize to new intents. recent work has focused on performing zero-shot intent detection on unseen intents and domains. using knowledge from ontologies or attributes @xcite can help in detecting and generalizing to new intents. a more recent approach by @xcite makes modifications to capsule networks to generalize to unseen domains. embeddings of intent descriptions have also shown to be quite meaningful in generalizing to new intents and services @xcite . while these methods are effective, they all require training on an initial set of intents. large language models (llms) like gpt-3 @xcite and more recently instruction finetuned models like @xcite have shown good zero-shot performance on newly seen tasks without any prior training data on those tasks. in this work, we show that these models are also effective for zero-shot intent classification using just intent descriptions. ## datasets we use public and private intent classification datasets to benchmark different approaches. for evaluation on public dataset, we use the english train and test sets from massive for intent classification. massive contains utterances directed at a physical device spanning 60 intents and 18 domains. for more details on the massive dataset @xcite , we encourage readers to refer to their paper. we also use private benchmarking datasets internal to our company. these datasets contain various intents and utterances in the enterprise setting spanning 3 different domains: it service management (itsm), hr and customer service management (csm). the utterances are inspired by interactions between humans and chatbots and are typically queries from goal-oriented conversations where the user needs to resolve an issue. additionally, some of these datasets also contain out-of-scope (oos) utterances in their test set i.e. utterances that do not belong to any intent, in order to benchmark irrelevance detection of intent classification models. ## methodology in this section, we describe the various methods we evaluate for zero and few-shot learning. ## observations comparing results across the 4 approaches, we notice that all 4 approaches are effective in low resource settings. we find that domain adaptation is a cheap option in terms of size of the models but it still requires 5-10 training utterances per intent for getting accuracy above 70%. data augmentation using paraphrasing further helps in most cases by 2-4 percentage points. however, expanding to new domains requires sentence-pairs data for training the sentence encoder which can involve days of human labeling. zero shot classification using intent descriptions with llms and instruction finetuned models performs even better than domain adaptation with data augmentation and doesn't require any utterances to be configured per intent. however a good description for each intent is required. additionally, these models can be expensive to operationalize. inference on flan-t5-xxl requires using a100 gpus. gpt-3 is not open-source and based on a pricing model which can be expensive to scale to thousands of customers. parameter efficient fine-tuning (peft) of instruction finetuned models like flan-t5-xl and flan-t5-large offers the best performance across all methods and often by a large margin. moreover, these models are only a fraction of the size of gpt-3 and flan-t5-xxl and much easier to operationalize at scale with far lesser compute resources. ## conclusion in this paper, we addressed the task of zero and few-shot intent identification using large language models (llms). we presented four approaches, namely domain adaptation, data augmentation, zero-shot prediction with prompting, and parameter-efficient fine-tuning. our experimental results demonstrate that llms and larger instruction fine-tuned language models are very effective in zero-shot setting with in-context prompting. smaller instruction finetuned models with adapters are even better when adapter-finetuned on just 1 or 3 examples per intent. we hope these results are useful for practical deployment of conversational agents in low-resource settings as well as aiding non-practitioners in building their intent classification models. in the future, we plan to extend this work by domain adapting smaller instruction finetuned models in a multi-task setting and exploring their zero-shot capabilities.