Generate then Refine: Data Augmentation for Zero-shot Intent Detection (2024)

Abstract: In this short paper we propose a data augmentation method for intent detection in zero-resource domains.Existing data augmentation methods rely on few labelled examples for each intent category, which can be expensive in settings with many possible intents.We use a two-stage approach: First, we generate utterances for intent labels using an open-source large language model in a zero-shot setting. Second, we develop a smaller sequence-to-sequence model (the Refiner), to improve the generated utterances. The Refiner is fine-tuned on seen domains and then applied to unseen domains. We evaluate our method by training an intent classifier on the generated data, and evaluating it on real (human) data.We find that the Refiner significantly improves the data utility and diversity over the zero-shot LLM baseline for unseen domains and over common baseline approaches.Our results indicate that a two-step approach of a generative LLM in zero-shot setting and a smaller sequence-to-sequence model can provide high-quality data for intent detection.

## introduction intent detection is a common component in taskoriented dialogue (tod) systems. its objective is to categorize user utterances into predefined classes of user intents @xcite . the advent of transformer-based models has significantly elevated the performance of intent detection, particularly when trained and evaluated within familiar domains and intents @xcite . however, when faced with previously unseen domains and intents, a considerable challenge emerges, primarily stemming from data scarcity. addressing this challenge involves maximizing the utility of limited training data and ensuring the adaptability of the trained intent classifier to novel intents. a combination of few-shot learning and 1 our code is available at: https://github.com/ tom192180/generate_then_refine meta-learning strategies has been employed in prior work @xcite @xcite . a viable alternative solution for low-resource domain adaptation is augmenting and increasing the training data. this is achieved by direct utilization of a generative large language model (llm) as a data generator @xcite @xcite @xcite . synthetically generated data, however, is not always of high quality and this has direct effect on model performance. as a remedy, several works have attempted to filter out generated examples that are of low quality or relevance @xcite @xcite . in this paper, we study the setup where no labelled data is available for unseen domains (e.g., travel), while substantial number of user utterances with intent labels exist for seen domains (e.g., banking). our goal is to create high-quality training data to train an intent classifier for completely unseen domains. in these unseen domains, the intent labels are known, but there are no utterances available. in line with previous work @xcite @xcite , we obtain generated data from a generative llm. the utility of this data, as measured by classification quality, lags behind a classifier trained on human utterances. one of the causes is that the llm tends to generate the words from the intent label in the output utterances, which makes it less representative of real user's utterances. therefore, we propose a refiner model that transforms sub-optimal generated utterances into better ones. the goal of our refiner model, trained on data from seen domains, is to enhance the performance of the intent classifier; see figure 1 for illustration. the main contribution of this paper is proposing a sequence-to-sequence learning method for enhancing the utility of llm-generated utterances, while retaining sample size. this stands in contrast to the previous studies that utilize formula-based metrics for data selection to filter sub-optimal utterances @xcite . in summary, the contributions of this paper are: ## related work data augmentation is a commonly used solution to data scarcity in training conversational agents @xcite . recently, generating new examples directly from llms has emerged as a viable strategy @xcite @xcite @xcite . in addressing the challenge of data scarcity for intent detection in novel domains, some of the prior work has applied metric-based meta-learning algorithms @xcite @xcite . data augmentation is an alternative. @xcite employ extrapolation techniques with a sequence-to-sequence model to generate new, under-represented utterances. alternatively, leveraging the advancements in llms, sahu et al. (2022) use gpt-3 with in-context learning to get additional utterances. lin et al. (2023) adopted a similar appraoch, using llms to generate synthetic examples, and then selecting useful data for training based on the pointwise v-information metric. our approach differs from previous research in two key ways. firstly, we specifically consider the zero-shot (instead of the few-shot) setting. this increases the challenge for llms to provide highquality data. secondly, rather than relying on a formula-based metric, we opt for a potentially more flexible approach -the refiner. the refiner directly generates refined data instead of filtering out irrelevant samples. ## experimental settings datasets. we use the clinc150 @xcite and sgd @xcite datasets for our experiments. clinc150 includes 10 domains, with a total of 150 intents (15 per domain). each intent has 100 training examples and 30 test examples. sgd includes 20 domains, with a total of 46 intents. intents in sgd have a diverse number of utterances, ranging from hundreds to thousands. the domains included in both datasets are detailed in appendix a. llms for utterances generation. we experiment with two llms to generate utterances: zephyr-7b beta @xcite and llama3-8b-instruct (ai@meta, 2024). an example of the prompt we use for zero-shot utterance generation is shown in figure 1 (top left). training the refiner. flan-t5-large (chung et al., 2024) with 783m parameters serves as the backbone model for the refiner. pairs of seendomain utterances and llm-generated utterances are used to train the refiner, which is trained for 6 epochs with a batch size of 24 and early stopping to prevent overfitting. during training, validation loss is monitored using a fine-tuned distilbert @xcite foot_0 classifier (67m parameters), trained on data from all seen domains. the classifier is fine-tuned over 1,800 steps with a batch size of 60, also using early stopping to avoid overfitting. the prompt we use for the refiner is: the sentences above are user intent expressions for "{intent}" in the "{do-main}" context, but they might have less quality or contain mistakes. provide one improved expression. here, "{intent}" and "{domain}" are placeholders derived from the datasets. ## ablation study effect of fine-tuning on performance. the results in factors to improve the training efficiency and performance. we additionally conduct an ablation study to explore various factors that could potentially affect training efficiency and performance: 1) we apply lora fine-tuning @xcite with only around 5m trainable parameters to see if efficient fine-tuning can achieve similar outcomes. the results are in ## conclusions we found that (1) our proposed utterance refiner can improve data quality on zero-resource domains without the need for larger llms or oversampling;