Data Augmentation for Intent Classiﬁcation with Off-the-shelf Large Language Models
Abstract
Data augmentation is a widely employed technique to alleviate the problem of data scarcity.
In this work, we propose a prompting-based approach to generate labelled training data for intent classiﬁcation with off-the-shelf language models (LMs) such as GPT-3.
An advantage of this method is that no task-speciﬁc
LM-ﬁne-tuning for data generation is required; hence the method requires no hyper-parameter tuning and is applicable even when the available training data is very scarce.
We evaluate the proposed method in a few-shot setting on four diverse intent classiﬁcation tasks.
We ﬁnd that GPT-generated data signiﬁcantly boosts the performance of intent classiﬁers when intents in consideration are sufﬁciently distinct from each other.
In tasks with semantically close intents, we observe that the generated data is less helpful.
Our analysis shows that this is because GPT often generates utterances that belong to a closely-related intent instead of the desired one.
We present preliminary evidence that a prompting-based GPT classiﬁer could be helpful in ﬁltering the generated data to enhance its quality.
Introduction
A key challenge in creating task-oriented conversational agents is gathering and labelling training data.
Standard data gathering options include manual authoring and crowd-sourcing.
Unfortunately, both of these options are tedious and expensive.
Data augmentation is a widely used strategy to alleviate this problem of data acquisition.
There are two particularly promising paradigms for data augmentation in natural language processing that use pretrained language models (LMs) (Peters et al., 2018; Devlin et al., 2018).
The ﬁrst family of methods ﬁne-tunes an LM on task-speciﬁc
Input Prompt:
The following sentences belong to the same category music_likeness:
Example 1: i like soft rock music
Example 2: current song rating three stars
Example 3: save this song as a favorite
Example 4: remind me that i like that song
Example 5: save my opinion on the currently playing song
Example 6: i love the song do you
Example 7: add the song to my favorites
Example 8: store opinion on song
Example 9: the song in background is cool
Example 10: i am the living blues
Example 11:
Completions: i dislike classical music she is a music lover i am a lover of painting this is the best song ever video that looks like the other video save preference on my profile express negative opinion on the song i am a great blues follower
the song is better than i thought this song is also fun data and generates new examples using the ﬁnetuned LM (Wu et al., 2018; Kumar et al., 2019, 2021; Anaby-Tavor et al., 2020; Lee et al., 2021).
A limitation of these methods is that, in a realworld scenario, task-speciﬁc data is scarce and ﬁnetuning an LM can quickly become the bottleneck.
The second family of methods sidesteps this bot- tleneck by employing off-the-shelf pretrained LMs such as GPT-3 (Brown et al., 2020) to directly generate text without any task-speciﬁc ﬁne-tuning.
In particular, data generation by the GPT3Mix approach by Yoo et al.
(2021) boosts performance on multiple classiﬁcation tasks; however, they only consider tasks with few (up to 6) classes and easyto-grasp class boundaries (e.g., positive and negative ).
This work studies the applicability of massive off-the-shelf LMs, such as GPT-3 and GPT-J (Wang and Komatsuzaki, 2021) to perform effective data augmentation for intent classiﬁcation (IC) tasks.
In
IC, the end goal is to predict a user’s intent given an utterance, i.e., what the user of a task-oriented chatbot wants to accomplish.
Data augmentation for IC is particularly challenging because the generative model must distinguish between a large number (in practice up to several hundreds) of ﬁne-grained intents that can be semantically very close to each other.
Prior methods such as GPT3Mix prompt the model with the names of all classes as well as a few examples from randomly chosen classes.
We test GPT3Mix for one and observe that such approaches are poorly suitable for intent classiﬁcation tasks with tens or hundreds of possible intents.
Instead, in this study, we use a simple prompt structure that focuses on a single seed intent (see Figure 1) as it combines the intent’s name and all available examples.
Our experiments primarily focus on few-shot IC on four prominent datasets: CLINC150 (Larson et al., 2019), HWU64 (Xingkun Liu and Rieser, 2019), Banking77 (Casanueva et al., 2020), and
SNIPS (Coucke et al., 2018).
We also consider a partial few-shot setup to compare to the Example
Extrapolation (Ex2) approach by Lee et al.
(2021) who use a similar prompt but ﬁne-tune the LM instead of using it as is.
The main ﬁndings of our experiments are as follows: (1) GPT-generated samples boost classiﬁcation accuracy when the considered intents are well-distinguished from each other (like in CLINC150, SNIPS).
(2) On more granular datasets (namely HWU64 and Banking77), we ﬁnd that GPT struggles in distinguishing between different confounding intents.
A smallscale study to further understand this behaviour suggests that GPT could be used as a classiﬁer to ﬁlter out unfaithful examples and enhance the quality of the generated training set.
Additionally, we investigate how valuable the generated data could be if relabelled by a human.
Using an oracle model, we show that (4) the human labelling of
GPT-generated examples can further improve the performance of intent classiﬁers, and that (5) LMgenerated data has a higher relabelling potential compared to edit-based augmentation techniques, such as Easy Data Augmentation (EDA) (Wei and
Zou, 2019).
Method
We consider training an intent classiﬁer, where an intent is a type of request that the conversational agent supports; e.g. the user may want to change the language of the conversation, play a song, transfer money between accounts, etc.
However, collecting many example utterances that express the same intent is difﬁcult and expensive.
Therefore, this paper experiments with a straightforward method to augment the training data available for an intent: creating prompts from the available examples and feeding them to a large language model such as
GPT-3 (Brown et al., 2020).
Figure 1 illustrates the process of data generation for an intent with K available examples.
Experimental Setup 3. Datasets
We use four intent classiﬁcation datasets in our experiments with varying levels of granularity among intents.
CLINC150 (Larson et al., 2019),
HWU64 (Xingkun Liu and Rieser, 2019) are multidomain datasets, each covering a wide range of typical task-oriented chatbot domains, such as playing music and setting up alarms.
Importantly, the CLINC150 task also contains examples of out-of-scope (OOS) utterances that do not correspond to any of CLINC’s 150 intents.
Banking77 (Casanueva et al., 2020) is a single domain dataset with very ﬁne-grained banking-related intents.
Finally, the SNIPS (Coucke et al., 2018) dataset contains 7 intents typical for the smart speaker usecase.
We refer the reader to Tablefor exact statistics of all used datasets.
3. Setup
The main data-scarce setup that we consider in this work is the few-shot setup , where only K =training examples are available for every intent of interest.
Additionally, to compare to example extrapolation with ﬁne-tuned language models as proposed by Lee et al.
(2021), we consider a partial
[Table]
few-shot setup . In the latter setting, we limit the amount of training data only for a handful of fewshot intents 2 and use the full training data for others.
When data augmentation is performed, we augment the few-shot intents to have N examples, where N is the median number of examples per intent of the original data.
To precisely describe the training and test data in all settings, we will use D part to refer to dataset parts, i.e. train, validation, and test.
In addition, we use D F and D M to refer to data-scarce and data-rich intents (the latter only occur in the partial few-shot setting).
This notation is deﬁned for all parts, therefore, D part
= D { F,part } ∪ D { M,part } , ∀ part ∈ { train, val, test } .
When GPT-3 or a baseline method is used to augment the training data we generate N − K examples per intent and refer to the resulting data as ˜ D F,train .
We experiment with four different-sized GPT-3 models 3 by
OpenAI and GPT-J by EleutherAI 4 to obtain ˜ D .
The four GPT-3 models are: Ada, Babbage, Curie, and Davinci.
In order, Ada is the smallest model and Davinci is the largest.
Model sizes of GPTengines are not known precisely but are estimated by Eleuther AI to be between 300M and 175B parameters 5 .
3. Training and Evaluation
We ﬁne-tune BERT-large (Devlin et al., 2018) on the task of intent classiﬁcation by adding a linear layer on top of the [CLS] token (Wolf et al., 2019).
In all setups we use the original validation set for tuning the classiﬁer’s training hyperparameters.
We chose to use the full validation set as opposed to a few-shot one to avoid issues with unstable hyperparameter tuning and focus on assessing the quality of the generated data.
Full few-shot.
In this setup, we treat all the intents as few-shot and evaluate our method on the following three scenarios: (i) Baseline : all the intents are truncated to K = 10 samples per intent, (ii) Augmented :
˜ D { F,train } is generated using GPT and models are trained on D { F,train } ∪ ˜ D { F,train } and (iii) EDA-baseline : same as above, but ˜ D { F,train } is generated using Easy Data Augmentation (EDA) by Wei and Zou (2019).
For each scenario, we report the 1) overall in-scope accuracy on the complete test set D test , i.e. intent classiﬁcation accuracy excluding OOS samples in the test set, and 2) few-shot classiﬁcation accuracy of the models on D { F,test } .
For CLINC150, we also report out-of-scope recall (OOS recall) on D test that we compute as the percentage of OOS examples that the model correctly labelled as such.
The purpose of this setting is to estimate what further gains can be achieved if the data generated by GPT were labelled by a human.
We train an oracle O on the full training data D train .
We also use O to assess the quality of the generated data.
Namely, we compute ﬁdelity of the generated data as the ratio of generated utterances that the oracle labels as indeed belonging to the intended seed intent.
A higher ﬁdelity value means that the generated samples are more faithful to original data distribution.
Partial few shot
In this setup, we train S intent classiﬁers, choosing different few-shot intents every time to obtain D F .
We then average the metrics across these S runs.
For CLINC150, S = 10 corresponding to the 10 different domains, whereas for SNIPS, S = 7 corresponding to the 7 different intents.
We evaluate our method on the following three scenarios introduced by Lee et al.
(i) Baseline : models are trained without data augmentation on D { F,train } ∪ D { M,train } .
(ii) Upsampled : D { F,train } is upsampled to have N examples per intent.
Then models are trained on upsampled
[Figure]
F e w - s h
o t
A cc u r a c y adababbagecuriedavinci
[Figure]
2.0 Temp.
OO S R e c a ll adababbagecuriedavinci
[Figure]
F i de li t y
[Figure]
D { F,train } ∪ D { M,train } . (iii) Augmented : models are trained on D { F,train } ∪ ˜ D { F,train } ∪ D { M,train } . For each scenario in this setup, we report the overall in-scope classiﬁcation accuracy (and OOS Recall for CLINC150).
For both partial few-shot and full few-shot settings, we report means and standard deviations over 10 repetitions of each experiment.
Experimental Results
Full few-shot.
Table 2 shows the results of our few-shot experiments.
For CLINC150 and SNIPS, data augmentation with GPT-3 is very effective as it leads to respective accuracy improvements of up to approximately 3.7% and 6% on these tasks.
These improvements are larger than what the baseline
EDA method brings, namely 2.4% and 2.9% for
[Figure]
CLINC150 and SNIPS.
Importantly, using larger
GPT models for data augmentation brings signiﬁcantly bigger gains.
Data augmentation results on
Banking77 and HWU64 are, however, much worse, with no or little improvement upon the plain fewshot baseline.
We present a thorough investigation of this behaviour in Section 4.1.
One can also see that data augmentation with GPT models lowers the OOS recall.
Next, we observe that relabelling EDA and GPTgenerated sentences by the oracle gives a significant boost to accuracies across the board, conﬁrming our hypothesis that human inspection of generated data could be fruitful.
Importantly, we note that the magnitude of improvement for EDA is less than for GPT models.
This suggests that
GPT models generate more diverse data that can eventually be more useful after careful human inspection.
Lastly, relabelling also improves OOS recall on CLINC150, which is due to the fact that much of the generated data was labelled as OOS by the oracle.
Partial few-shot.
Table 3 shows the results of our partial few-shot experiments on CLINC and SNIPS.
By augmenting the dataset with GPT- [Table] [Table] generated samples, the few-shot accuracy improves by up to 2.92% on CLINC150 and 18.14% on
SNIPS compared to the baseline setting.
Our method achieves competitive results compared to
Ex2 (Lee et al., 2021), both in terms of absolute accuracies and the relative gains brought by data augmentation.
Note that Ex2 uses T5-XL (Roberts et al., 2020) with nearly 3 billion parameters as its base intent classiﬁer, while our method uses
BERT-large with only 340 million parameters.
4. Analysis Effect
of GPT sampling temperature.
We investigate the impact of generation temperature on the quality and ﬁdelity of generated data.
We perform this investigation on the CLINC150 dataset using the partial few-shot setup.
Results in Figure 2 show that, for all engines, the generated data leads to the
Davinci generated sentences
Seed Intent
Oracle Prediction
HWU play a song with the word honey
music_likeness play_music you are playing music
music_likeness play_music ‘let me hear some of that jazz!’
music_likeness play_music i really like myspace music play_music
music_likeness i love the start lucky country music play_music music_likeness thank you for the music play_music music_likeness please play the next song music_settings play_music play background music music_settings play_music play the hour long loop of rock song music_settings
play_music need you to play that song one more time play_music music_settings skip that song, its turkish play_music music_settings pickup the beat or a temp track or audio plugin play_music music_settings
Banking
My last attempt to top up didn’t seem to work , any success?
topping_up_by_card top_up_failed
I tried to top off my wallet using my card but it says “top up failed".
topping_up_by_card top_up_failed
I cannot top-up by my cellular phone number?
How do I do that?
topping_up_by_card top_up_failed
Can you transfer money to my Ola prepaid option?
Or help me top up my card to money .
They never accept my card
so I always have to suffer top_up_failed topping_up_by_card
Hi my app is activated on activate.co.in, but unable to top up my phone.
I tried credit card, debit card and Paytm but fails top_up_failed topping_up_by_card
I try to top up my card
but it’s not going through.
It’s still on pending status.
Do I need to wait or did I do something wrong top_up_failed pending_top_up
I tried top-up with my card but notiﬁcation shows that ‘Pending’.
This has been happening since last night.
Can you tell me what’s going on top_up_failed pending_
[Table]
highest classiﬁcation accuracy when the generation temperature is around 1.0, although lower temperatures result in higher OOS recall.
We also observe that the ﬁdelity of the generated samples decreases as we increase the temperature (i.e. higher diversity, see Figure 2c).
This suggests that higher ﬁdelity does not always imply better quality samples as the language model may simply copy or produce less diverse utterances at lower temperatures.
In Appendix A, we perform a human evaluation, reaching similar conclusions as when using an oracle to approximate ﬁdelity.
Fidelity on different datasets.
Our results in Section 4 show that data augmentation gains are much higher on CLINC150 and SNIPS than on HWU and Banking77.
To contextualize these results, we report the ﬁdelity of GPT-J-generated data for all these tasks in Figure 3b.
Across all generation temperatures, the ﬁdelity of the generated data is higher for CLINC150 and SNIPS than for HWU64 and
Banking77.
For all datasets, the ﬁdelity is higher when the generation temperature is lower; however,
Figure 3a shows that low-temperature data also does improve the model’s performance.
Data generation for close intents.
To better understand the lower ﬁdelity and accuracy on
HWU64 and Banking77 datasets, we focus on intents with the lowest ﬁdelities.
Here, by intent ﬁdelity, we mean the percentage of the intent’s generated data that the oracle classiﬁed as indeed belonging to the seed intent.
In the Banking77 dataset, the lowest-ﬁdelity intent is “topping_up_by_card.”
For this intent, only 33% of the Davinci-generated sentences were labelled as “topping_up_by_card”
Fidelity (3 intents)
HWU
Banking w/o ﬁltering (468) 60.
w/ ﬁltering (371) 72.
3-way accuracy
Davinci 86.
10-shot BERT-large 82.
Full data BERT-large 94.
[Table]
[Figure]
by the oracle, implying that two-thirds of the sentences did not ﬁt that intent, “top_up_failed" and “top_up_card_charge" being the two most common alternatives chosen by the oracle.
Similarly, only 50% of the Davinci-generated sentences abide by the lowest-ﬁdelity “music_likeness" intent in the HWU64 dataset, “play_music” and “general_quirky" being the most common intents among the “unfaithful" sentences.
Figure 4 visualizes this high percentage of unfaithful generated sentences.
It also shows the proportion of the two most common alternatives that the oracle preferred over the seed intent.
Table 4 presents generated sentences for confounding intents in the HWU and Banking77 datasets.
There are clear indications of mix-up of intents, e.g., Davinci generates, “play a song with the word honey," which should belong to “play_music" rather than “music_likeness."
There are also instances where the LM mixes two intents; for instance, Davinci generates “Hi my app is activated on activate.co.in, but unable to top up my phone.
I tried credit card, debit card and Paytm but fails," which could belong to either “topping_up_by_card" intent (as it mentions about using credit card in the context of a top up) or “top_up_failed" (as the top up ultimately fails).
4. Can GPT Models Understand Close Intents?
We perform extra investigations to better understand what limits GPT-3’s ability to generate data accurately.
We hypothesize that one limiting factor can be GPT-3’s inability to understand ﬁne-grained differences in the meanings of utterances.
To verify this hypothesis, we evaluate how accurate GPT-3 is at classifying given utterances as opposed to generating new ones.
Due to the limited prompt size of 2048 tokens, we can not prompt GPT-3 to predict all the intents in the considered datasets.
We thus focus on the close intent triplets from HWU64 and
Banking77 datasets that we use in Table 4.
We compare the 3-way accuracy of a prompted GPTclassiﬁer to the similarly-measured 3-way performance of conventional BERT-large classiﬁers.
We prompt GPT-3 with 10 examples per intent (see
Figure 5).
For comparison, we train BERT-large classiﬁers on either the same 10 examples or the full training set.
Table 5 shows that the Davinci version of GPT-3 performs in between the 10-shot and the full-data conventional classiﬁers.
This suggests that while GPT-3’s understanding of nuanced intent differences is imperfect, it could still be sufﬁcient to improve the performance of the downstream few-shot model.
Inspired by this ﬁnding, we experiment with using GPT-3’s classiﬁcation abilities to improve the quality of generated data.
Namely, we reject the generated utterances that GPT-3 classiﬁes as not belonging to the seed intent.
For both
HWU64 and Banking77, this ﬁltering method signiﬁcantly improves the ﬁdelity of the generated data for the chosen close intent triplets. 4. Comparison with GPT3Mix
To test our initial hypothesis that prior methods such as GPT3Mix are not suitable for intent classiﬁcation, we experiment with the said method on the CLINC150 dataset using Curie. Speciﬁcally, we include an enumeration of the 150 intent names in the prompt and randomly select one example for
K intents.
We observe a poor in-scope accuracy of 86.33% in the Augmented scenario 6 .
Furthermore, the generated samples have low ﬁdelity (27.96%). We also test a mixture of GPT3Mix prompt and our prompt where we include all the K examples for the seed intent instead of 1 example per K randomly sampled intents. This mixed variant also performs poorly on CLINC150 and only achieves an in-scope accuracy of 86.05% 7 and a ﬁdelity of 33.56%. Our interpretation of this result is that
GPT cannot handle the long list of 150 intent names in the prompt.
Related Work
The natural language processing literature features diverse data augmentation methods.
Edit-based methods such as Easy Data Augmentation apply rule-based changes to the original utterances to produce new ones (Wei and Zou, 2019).
In backtranslation methods (Sennrich et al., 2016) available examples are translated to another language and back.
Recently, data augmentation with ﬁnetuned LMs has become the dominant paradigm (Wu et al., 2018; Kumar et al., 2019, 2021;
AnabyTavor et al., 2020; Lee et al., 2021).
Our simpler method sidesteps LM-ﬁne-tuning and directly uses off-she-shelf LMs as is.
The data augmentation approach that is closest to the one we use here is GPT3Mix by Yoo et al.
A key part of the GPT3Mix prompt is a list of names of all possible classes (e.g. “The sentiment is one of ‘positive’ or ‘negative”’).
The
LM is then expected to pick a random class from the list and generate a new example as well as the corresponding label.
However, this approach does not scale to intent classiﬁcation setups, which often
Input Prompt:
Each example in the following list contains a sentence that belongs to a category. A category is one of the following: music_likeness, play_music, music_settings: sentence: next i want to hear shinedown ; category: play_music sentence: i am the living blues ; category: music_likeness sentence: open music player settings ; category: music_settings sentence: play hopsin from my latest playlist ; category: play_music sentence: i like this song ; category: GPT-3 Predictions: play_music,music_likeness,music_settings, music_likeness,music_likeness,help_command feature hundreds of intents (see Section 4.3). Therefore, we choose a different prompt that encourages the model to extrapolate between examples of a seed intent similarly to (Lee et al., 2021).
Other work on few-shot intent classiﬁcation explores ﬁne-tuning dialogue-speciﬁc LMs as classiﬁers as well as using similarity-based classiﬁers instead of MLP-based ones on top of BERT (Vuli ´ c et al., 2021).
We believe that improvements brought by data augmentation would be complementary to the gains brought by these methods.
Lastly, our method to ﬁlter out unfaithful GPT generations is related to the recent work by Wang et al.
(2021) that proposes using GPT3 for data labelling.
A crucial difference with respect to our work, however, is that we use GPT-3 for rejecting mislabelled samples rather than proposing labels for unlabelled samples.
Conclusion
We propose a prompt-based method to generate intent classiﬁcation data with large pretrained lan- guage models.
Our experiments show that generated data can be helpful as additional labelled data for some tasks, whereas, for other tasks, the generated data needs to be either relabelled or ﬁltered to be helpful.
We show that a ﬁltering method that recasts the same GPT model as a classiﬁer can be effective.
Our ﬁltering method, however, requires knowing the other intents that the generated data is likely to belong to instead of the seed intent.
Future work can experiment with heuristics for approximately identifying the most likely actual intents for the generated utterances.
This would complete a data generation and ﬁltering pipeline that, according to our preliminary results in Section 4.
here, could be effective.
Other ﬁltering methods could also be applied, such as looking at the likelihood of the generated utterances as explored in a concurrent work by Meng et al.
Lastly, an interesting future work direction is identifying which generated utterances most likely need a human inspection.  Ethical Considerations
As discussed for the GPT3Mix method in Yoo et al.
(2021), using large language models for data augmentation presents several challenges: they exhibit social biases and are prone to generating toxic content. Therefore, samples generated using our prompting-based approach need to be considered carefully. To address such ethical concerns, human inspection would be the most reliable way to identify and ﬁlter out problematic generations. The practitoners who apply our method may also consider debiasing the language model before using it for generation (Schick and Schütze, 2021).
Appendix A Human Evaluation
In Figure 2 we evaluate the ﬁdelity of the samples generated by GPT-3 with respect to the original set of sentences used to prompt it.
Fidelity is approximated by the classiﬁcation performance of an "oracle" intent classiﬁer trained on the whole dataset ( D train ∪ D test ) and evaluated over the generated samples.
In order assess whether the oracle predictions are comparable to those of a human, we perform a human evaluation study.
[Figure]
We consider that a model produces sentences with high ﬁdelity if a human is unable to distinguish them from a set of human-generated sentences belonging to the same intent.
Therefore, for each intent in the CLINC150 dataset, we sample ﬁve random examples and we randomly choose whether to replace one of them by a GPT-3 generated sentence from the same intent.
We generate sentences with each of the four GPT-3 models considered in the main text with two different temperatures (0.8 and 1.0).
The sentence to replace is randomly selected.
Finally, the ﬁve sentences are displayed to a human who has to choose which of the sentences is generated by GPT-3, if any.
The task is presented to human evaluators in the form of a web application (see Figure 7).
We placed a button next to each sentence in order to force human evaluators to individually consider each of the examples.
Once annotated, the evaluator can either submit , discard , or leave the task to label later .
We used a set of 15 voluntary evaluators from multiple backgrounds, nationalities, and genders.
Each evaluator annotated an average ofexamples, reaching a total of 500 evaluated tasks.
For each model and temperature, we report the error rate of humans evaluating whether a task contains a GPT-generated sample.
We consider that evaluators succeeds at a given task when they correctly ﬁnd the sentence that was generated by GPT or when they identify that none of them was generated.
Thus, the error rate for a given model and temperature is calculated as #failed / total_evaluated.
Results are displayed in Figure 6.
We ﬁnd that human evaluators tend to make more mistakes when the temperature used to sample sentences from GPT-3 is smaller.
This result is expected since lowering the temperature results in sentences closer to those prompted to GPT-3, which are humanmade.
We also observe that models with higher capacity such as Davinci tend to generate more indistinguishable sentences than lower-capacity models such as Ada , even for higher temperatures.
These results are in agreement with the "oracle" ﬁdelity results introduced in Figure 2.