P rompt M ix: A Class Boundary Augmentation Method for Large Language Model Distillation (2023)

Abstract: Data augmentation is a widely used technique to address the problem of text classification when there is a limited amount of training data. Recent work often tackles this problem using large language models (LLMs) like GPT3 that can generate new examples given already available ones. In this work, we propose a method to generate more helpful augmented data by utilizing the LLM’s abilities to follow instructions and perform few-shot classifications. Our specific PromptMix method consists of two steps: 1) generate challenging text augmentations near class boundaries; however, generating borderline examples increases the risk of false positives in the dataset, so we 2) relabel the text augmentations using a prompting-based LLM classifier to enhance the correctness of labels in the generated data. We evaluate the proposed method in challenging 2-shot and zero-shot settings on four text classification datasets: Banking77, TREC6, Subjectivity (SUBJ), and Twitter Complaints. Our experiments show that generating and, crucially, relabeling borderline examples facilitates the transfer of knowledge of a massive LLM like GPT3.5-turbo into smaller and cheaper classifiers like DistilBERT-base and BERT-base. Furthermore, 2-shot PromptMix outperforms multiple 5-shot data augmentation methods on the four datasets. Our code is available at https://github.com/ServiceNow/PromptMix-EMNLP-2023.

## introduction data scarcity is a key challenge in numerous reallife scenarios, such as deploying intent detection systems in task-oriented conversational agents and identifying hateful instances of speech on social media platforms. crowdsourcing has been a traditionally popular choice to obtain additional data, but it is a financially expensive procedure incurring a high cost of human labor @xcite ; promptmix prompts w/o mixup eda seed ex. eda prompts w/o mixup seed ex. figure 1 : promptmix focuses on generating examples near the class boundary of two classes, unlike other standard augmentation approaches like eda @xcite and prompting-based methods without mixup @xcite , that only use the information of a single class for augmentation. @xcite @xcite . with the recent surge in the development of generative large language models (llms) @xcite @xcite , a large body of literature has emerged that employs llms to generate additional data for various tasks @xcite @xcite . in this work, we focus on the task of few-shot text classification @xcite @xcite . specifically, we explore zero-shot and 2-shot settings. early works employing llms to generate additional data samples for text classification first fine-tune a generative language model on an initial seed dataset and then use it to synthesize new training data @xcite @xcite ; however, the fine-tuning step quickly becomes a bottleneck in the absence of sufficient 1) age_limit is about customer inquiries on age-related restrictions for opening a bank account. some examples: . . 2) atm_support is about users asking how to use an atm, where to find one, or any other clarifications about a transaction at an atm. some examples: augmented dataset class: age_limit class: atm_support can my 13 year old son open a bank account and use his card at an atm? class: age_limit class: atm_support seed examples train class: age_limit step 1: generate class: atm_support step 2: relabel class: age_limit my teenage son's account charges a fee for atm withdrawals. can you help? seed examples. more recent works sidestep finetuning by designing natural language prompts for off-the-shelf llms @xcite @xcite . a key limitation of such works is that their prompts only focus on using information from a single class when generating augmentations. additionally, they do not incentivize the llm to diversify the generated examples, and recent works show that instruction-tuned llms like instructgpt @xcite and vicuna @xcite are prone to mode collapse @xcite . to address the previous limitations, we propose a two-step prompting-based technique, promptmix. first, promptmix instructs an llm (in our case, gpt3.5-turbo 1 ) to generate new examples by mixing information from multiple classes. the degree of mixing is controlled by a parameter α, and using a range of α values diversifies the generated examples; however, promoting mixup increases the risk of false positive generations, so promptmix uses a relabelling scheme in the second step to improve the faithfulness of the generated examples. in particular, it uses an llm as a classifier to assign new labels to the generated examples. we find that training a classifier on these relabeled examples effectively transfers the knowledge of a massive llm like gpt3.5 into much smaller models like bert @xcite and distilbert @xcite . figure 2 demonstrates the complete promptmix framework. we summarize our contributions as follows: a) we propose promptmix, a novel two-step prompting-based method to generate a diverse 1 https://platform.openai.com/docs/models/gpt-3-5 set of labeled examples for any text classification dataset; and b) we demonstrate that generating borderline examples and relabeling them improves knowledge transfer from a massive llm like gpt3.5 into much smaller models like dis-tilbert and bert, even without abundant seed examples. we also show that 2-shot promptmix outperforms multiple data augmentation baselines that use 5-shot or more seed data. ## related work our work intersects with the topics of data augmentation, few-shot classification, and knowledge distillation, which we explain in detail below. ## methodology we hypothesize that training a robust text classifier requires the training data to have a good mix of borderline examples @xcite . this section describes our method promptmix, where we first instruct an llm (gpt3.5-turbo, in our case) to generate difficult examples near class boundaries then relabel those generations to improve the faithfulness of the generated data (see figure 2 ). ## does promptmix generate borderline input classes age_limit (75%) atm_support (25%) w/ mixup -can someone under 18 open an account with unlimited atm withdrawal limit? -can i open an account for my teenage daughter ? -do i need to be over a certain age to use an atm? -can i use my children's debit card to withdraw money from the atm? w/o mixup -can i open an account for a minor? -what is the minimum age requirement to get a bank account? -is there an age restriction for account holders? -is there a minimum age requirement to open an account? ## results referring to ## conclusion to conclude, we propose promptmix, a novel two-step prompting-based method for generating borderline augmented examples in extreme fewshot text classification setups. our method combines the process of text augmentation, pseudolabeling, and knowledge distillation in a cohesive pipeline. we show that by generating borderline training examples and relabeling them using a large teacher model like gpt3.5-turbo, we can transfer the knowledge of such massive llms into much smaller models like distilbert base and bert base . furthermore, 2-shot promptmix beats multiple 5shot or higher data augmentation baselines, making it a highly promising data augmentation approach.