Selective In-Context Data Augmentation for Intent Detection using Pointwise V -Information (2023)

Abstract: This work focuses on in-context data augmentation for intent detection. Having found that augmentation via in-context prompting of large pre-trained language models (PLMs) alone does not improve performance, we introduce a novel approach based on PLMs and pointwise V-information (PVI), a metric that can measure the usefulness of a datapoint for training a model. Our method first fine-tunes a PLM on a small seed of training data and then synthesizes new datapoints - utterances that correspond to given intents. It then employs intent-aware filtering, based on PVI, to remove datapoints that are not helpful to the downstream intent classifier. Our method is thus able to leverage the expressive power of large language models to produce diverse training data. Empirical results demonstrate that our method can produce synthetic training data that achieve state-of-the-art performance on three challenging intent detection datasets under few-shot settings (1.28% absolute improvement in 5-shot and 1.18% absolute in 10-shot, on average) and perform on par with the state-of-the-art in full-shot settings (within 0.01% absolute, on average).

## introduction intent detection, defined as the identification of a user's intent given an utterance, is a fundamental element in task-oriented dialogue systems, usually occurring within the natural language understanding (nlu) component. one of the practical challenges of training and deploying nlu modules is data scarcity, due to various reasons, such as under-represented languages, privacy and ethical concerns, or simply the cost of collecting and annotating sufficiently large amounts of data for new intents. consequently, accurately identifying intents in limited-resource scenarios has drawn attention from the community @xcite @xcite , for example). there are three main families of approaches that address the challenge of limited data for intent detection: data augmentation @xcite , focusing on generating high-quality synthetic training and evaluation data; few-shot learning @xcite , focusing on creating learning algorithms that can cope with limited amounts of data; and transfer learning @xcite , focusing on learning algorithms that can generalize across domains (therefore not requiring in-domain data). in this work, we follow the data augmentation approach, which is a general method that attempts to augment a humanauthored dataset with a large set of syntheticallygenerated instances. most recent work has suggested using pre-trained language models (plms) for data augmentations under various setups, e.g., @xcite , showing great improvements in performance. however, simply generating a large number of synthetic data points is not enough; we need to consider the quality of each data point, i.e., how beneficial it would be to the model's performance if that synthetic data point is added to the training set. this is an important issue since the model might learn to overfit to synthetic datapoints (which may be low quality, represent specific use cases, etc.) and thus under-perform on real data. in this work, we propose to apply pointwise v-information (pvi) @xcite for data augmentation, in a way that leverages a plm to generate synthetic examples that are relevant and beneficial for training the downstream model, which in our case is an intent classifier. our contributions are as follows: the rest of the paper is organized as follows: in section 2 we present relevant work and in section 3 we introduce our method. in sections 4 and 5 we discuss training details, experiments, and results. in section 6, we present our analysis and discuss alternative approaches we investigated. in section 7 we conclude, and in the following sections we discuss limitations and ethical considerations. ## related work intent detection intent detection is the task of identifying the user's intent by mapping the user's natural language utterance into one of several predefined classes @xcite . it is a critical component in the pipeline of task-oriented dialogue systems, as it is used to determine the user's goal and to trigger an appropriate system action @xcite . several datasets have been proposed to evaluate the performance of intent detection models @xcite @xcite , for some recent examples). with the availability of such datasets, intent detection has been extensively studied in the literature. recently, pre-trained language models (e.g., bert @xcite ) have been shown to be effective in intent detection @xcite @xcite . ## in-context data augmentation in the following section, we describe our proposed two-stage method for data augmentation, which we refer to as in-context data augmentation (icda). the overall procedure is summarized in algorithm 1. we apply icda to the task of fewshot intent detection, which involves classifying a user utterance x into an intent label @xmath0. icda aims to generate synthetic examples x ′ such that they would belong to a given intent y. ## experimental results we conduct experiments on three benchmark datasets to validate the effectiveness of our proposed method. we first use opt-66b to generate augmentation examples and then apply our method to enhance a roberta-large model trained on three datasets. we repeat all experiments with 5 random seeds and report the average performance in full-shot and few-shot settings. to investigate the effect of the synthetic data size, we experiment with a variety of multipliers (see full-shot settings. in this setting, we use the entire training set for each domain. the proposed method achieves the best performance on banking and comparable results on hwu and clinc. in particular, on banking, we improve the convbert + combined baseline @xcite ) by 0.59% (absolute) and the roberta-large baseline by 0.72% (absolute). compared with the convbert + combined, which is pretrained on intent detection datasets in a selfsupervised fashion and adds examples-driven training and specific model architectural design, our method achieves similar results with much simpler model design. furthermore, our method is orthogonal to model architectures and can be integrated with any other approach for further improvement. we also find that icda improves the performance of the roberta-large model on hwu and clinc. this highlights the effectiveness of our method for enhancing intent detection models. ## analysis and discussion in this section, we analyze the performance of icda and other approaches we tried. we first identify several factors that affect performance, and then present evidence that icda works by transferring knowledge from the pretrained generator to the task model. we then discuss a data-relabelling experiment and an experiment using uncertainty measures or data cartography @xcite ## conclusion we introduced in-context data augmentation, a novel data augmentation framework to generate synthetic training data, preserving quality and diversity. we demonstrate that icda is effective on multiple intent detection benchmarks, with state-ofthe-art few-shot performance. our analysis shows that icda tends to perform better in low-resource settings and that our pvi filtering strategy is important for performance. future work includes applying icda to other conversational understanding tasks such as slot filling and dialogue state tracking, and incorporating other filtering or data selection strategies for further performance gains.