From Discrimination to Generation: Low-Resource Intent Detection with Language Model Instruction Tuning (2024)

Abstract: Intent detection aims to identify user goals from utterances, and is a ubiquitous step towards the satisfaction of user desired needs in many interaction systems. As dynamic and varied intents arise, models that are capable of identifying new intents promptly are required. However, existing studies usually fine-tune discriminative models on the specific defined intent classes, precluding them from being directly adopted to new intent domains. In this paper, we introduce a generative pre-trained intent model that can recognize new intents from different domains in low-resource scenarios. We reformulate intent detection into a generation task and design descriptive and regularized instructions to guide the model effectively to detect new intents in open domains with no parameter updates. To validate the proposed method, we introduce a new intent detection benchmark, including the Meta-Intent Dataset and three types of representative evaluation settings. We conduct extensive experiments which demonstrate that our method outperforms a range of strong baselines that needs further fine-tuning or domain-specific samples.

## introduction intent detection aims to identify user intentions or main topic, playing the first and foremost role in task-oriented dialogue systems @xcite , recommend systems @xcite and other ubiquitous systems. many existing studies have conducted considerable exploration @xcite , where pre-trained language models are fine-tuned under full supervision. however, due to the dynamic change of intents over * corresponding author. time, gathering a large number of manually annotated examples and pre-defining a comprehensive intent set are quite challenging. consequently, lowresource intent detection is proposed to recover new emerging intents with few or even zero labeled samples for relieving the need of substantial domain-specific labeled data. recent studies have been proposed to utilize meta-learning based methods, which transfer knowledge from seen intents to target new intents @xcite @xcite . also, @xcite propose to directly fine-tune the model on the new intents with context augmentation and self-distillation. as illustrated in figure 1a , these methods aim to fine-tune a customized model for each dataset. however, they require domain-specific labeled data for each intent detection task, which restricts the model generalization and leads to poor performance when facing large gap domains. another type of method is designed to learn a unified intent-aware encoder, as shown in figure 1b . @xcite propose intentbert, an intent utterance encoder which firstly is trained on public source corpora and then is adapted to target domains via non-parametric bank77 (banking domain) mcid (coronavirus corpus) how can i check if i have received a refund if it is not yet on my statement? what can i do to treat myself if i am sick with coronavirus? ## approach we introduce gen-pint: a generative pre-trained intent detection model via instruction tuning. as illustrated in figure 2 , gen-pint is a generative model with instructions. different from previous methods, gen-pint is trained on a collection of general intent detection corpora and is directly evaluated on unseen target datasets without parameter updates. at test phase, leveraging several demonstrated samples and task instructions, it learns to recognize semantic information in utterances and then generates the corresponding intent label from given options as the predicted result. ## conclusion in this paper, we reformulate intent detection into a generation task and introduce gen-pint, a new generative pre-trained model for low-resource intent detection. by conditioning on several training samples and instructions, gen-pint, as a unified and end-to-end model, is able to recover semantic of inputs and generate intents for utterances from different domains. we further build a new benchmark for intent detection with a diverse collections of datasets and conduct comprehensive evaluation under three settings. gen-pint outperforms a series of strong baselines including discriminative methods, in-context learning and prompt tuning. also, gen-pint performs better than recent dominated meta-learning methods and it can generate new descriptive intents that are beyond the reach of discriminative models.