Data Augmentation for Intent Classification with Off-the-shelf Large Language Models (2022)

Abstract: Data augmentation is a widely employed technique to alleviate the problem of data scarcity. In this work, we propose a prompting-based approach to generate labelled training data for intent classification with off-the-shelf language models (LMs) such as GPT-3. An advantage of this method is that no task-specific LM-fine-tuning for data generation is required; hence the method requires no hyper parameter tuning and is applicable even when the available training data is very scarce. We evaluate the proposed method in a few-shot setting on four diverse intent classification tasks. We find that GPT-generated data significantly boosts the performance of intent classifiers when intents in consideration are sufficiently distinct from each other. In tasks with semantically close intents, we observe that the generated data is less helpful. Our analysis shows that this is because GPT often generates utterances that belong to a closely-related intent instead of the desired one. We present preliminary evidence that a prompting-based GPT classifier could be helpful in filtering the generated data to enhance its quality.

## introduction a key challenge in creating task-oriented conversational agents is gathering and labelling training data. standard data gathering options include manual authoring and crowd-sourcing. unfortunately, both of these options are tedious and expensive. data augmentation is a widely used strategy to alleviate this problem of data acquisition. there are two particularly promising paradigms for data augmentation in natural language processing that use pretrained language models (lms) @xcite . the first family of methods fine-tunes an lm on task-specific ## method we consider training an intent classifier, where an intent is a type of request that the conversational agent supports; e.g. the user may want to change the language of the conversation, play a song, transfer money between accounts, etc. however, collecting many example utterances that express the same intent is difficult and expensive. therefore, this paper experiments with a straightforward method to augment the training data available for an intent: creating prompts from the available examples and feeding them to a large language model such as @xcite . figure 1 illustrates the process of data generation for an intent with k available examples. ## experimental results full few-shot. next, we observe that relabelling eda and gptgenerated sentences by the oracle gives a significant boost to accuracies across the board, confirming our hypothesis that human inspection of generated data could be fruitful. importantly, we note that the magnitude of improvement for eda is less than for gpt models. this suggests that gpt models generate more diverse data that can eventually be more useful after careful human inspection. lastly, relabelling also improves oos recall on clinc150, which is due to the fact that much of the generated data was labelled as oos by the oracle. partial few-shot. ## related work the natural language processing literature features diverse data augmentation methods. edit-based methods such as easy data augmentation apply rule-based changes to the original utterances to produce new ones @xcite . in backtranslation methods @xcite available examples are translated to another language and back. recently, data augmentation with finetuned lms has become the dominant paradigm @xcite @xcite @xcite . our simpler method sidesteps lm-fine-tuning and directly uses off-she-shelf lms as is. the data augmentation approach that is closest to the one we use here is gpt3mix by @xcite . a key part of the gpt3mix prompt is a list of names of all possible classes (e.g. "the sentiment is one of 'positive' or 'negative"'). the lm is then expected to pick a random class from the list and generate a new example as well as the corresponding label. however, this approach does not scale to intent classification setups, which often 6 average of 10 runs with a standard deviation of 1.17 7 average of 10 runs with a standard deviation of 0.59 ## conclusion we propose a prompt-based method to generate intent classification data with large pretrained lan-guage models. our experiments show that generated data can be helpful as additional labelled data for some tasks, whereas, for other tasks, the generated data needs to be either relabelled or filtered to be helpful. we show that a filtering method that recasts the same gpt model as a classifier can be effective. our filtering method, however, requires knowing the other intents that the generated data is likely to belong to instead of the seed intent. future work can experiment with heuristics for approximately identifying the most likely actual intents for the generated utterances. this would complete a data generation and filtering pipeline that, according to our preliminary results in section 4.2 here, could be effective. other filtering methods could also be applied, such as looking at the likelihood of the generated utterances as explored in a concurrent work by @xcite . lastly, an interesting future work direction is identifying which generated utterances most likely need a human inspection.