Self-Evolution Learning for Mixup: Enhance Data Augmentation on Few-Shot Text Classification Tasks (2023)

Abstract: Text classification tasks often encounter few-shot scenarios with limited labeled data, and addressing data scarcity is crucial. Data augmentation with mixup merges sample pairs to generate new pseudos, which can relieve the data deficiency issue in text classification. However, the quality of pseudo-samples generated by mixup exhibits significant variations. Most of the mixup methods fail to consider the varying degree of learning difficulty in different stages of training. And mixup generates new samples with one-hot labels, which encourages the model to produce a high prediction score for the correct class that is much larger than other classes, resulting in the model’s over-confidence. In this paper, we propose a self-evolution learning (SE) based mixup approach for data augmentation in text classification, which can generate more adaptive and model-friendly pseudo samples for the model training. SE caters to the growth of the model learning ability and adapts to the ability when generating training samples. To alleviate the model over-confidence, we introduce an instance-specific label smoothing regularization approach, which linearly interpolates the model’s output and one-hot labels of the original samples to generate new soft labels for label mixing up. Through experimental analysis, experiments show that our SE brings consistent and significant improvements upon different mixup methods. In-depth analyses demonstrate that SE enhances the model’s generalization ability.

## introduction recently, generative large language models (llms) have won great popularity in natural language processing (nlp), and have achieved impressive performance on various nlp tasks (kocoń * haoqi zheng and qihuang zhong contribute equally to this work. in text classification tasks, da methods can be divided into two categories: da methods like eda @xcite , back-translation @xcite , and others based on synthesis such as mixup. the first category conducts da by altering only the inputs. these methods only alter the inputs to generate new data while maintaining the original labels. these methods are easy to implement, but the input only changes a little thus leading to augmented inputs with limited diversity, which may reduce model generalization. the second category of da methods modify both inputs and labels, which changes the input samples in a certain way and simultaneously changes the corresponding labels to compose a new sample. these methods tend to generate samples more distinct from the original samples. mixup is a da method that modifies both inputs and labels. it mixes up inputs of samples and their labels, where labels are commonly represented with one-hot encoding. most of these methods mix up inputs of two samples on their input text @xcite or hidden-level representations @xcite . however, the pseudo sample, simply combined with two samples, may not be adaptive to the model's learning ability and friendly to the model training. recently, some work @xcite have focused on selecting similar sample pairs for the mixup. @xcite select samples according to the embedding similarity. park and caragea (2022) merge one sample considering the confidence of the model's predictions. moreover, in few-shot scenarios, using hard labels (one-hot labels) can lead to issues, where the one-hot labels fail to provide uncertainty of inputs since all the probability mass is given to one class. this results in over-confident models since the largest logit becomes larger than the others which removes the uncertainty of label space @xcite . the current label smoothing techniques generate soft labels that cannot dynamically adapt to the model's increasing ability as the training goes on, so they also cannot adjust according to the model's performance at the current stage. in this paper, we propose self-evolution learning for mixup to achieve data augmentation in text classification tasks. to cater to the model's learning ability, we first divide the training data into easyto-learn and hard-to-learn subsets. we gradually start from the mixup of easy-to-learn samples and then gradually transition to the mixup operation of hard-to-learn samples. to avoid the model's overconfidence, we introduce an instance-specific label smoothing method, where we linearly interpolate the predicted probability distribution of the original sample and its one-hot label to obtain a soft label. using this soft label reduces the difference between the model's prediction probability for different classes, which can alleviate the model's overconfidence. additionally, this instance-specific label can dynamically adapt to the growth of the model's increasing ability and can be customized to the model's current performance. our method has empirically proven that mixing up in the order of increased difficulty can make the generated samples more adaptive for model training compared to randomly selected samples. our contributions are as follows: ## conclusion in this paper, we propose a simple-yet-effective self-evolution (se) learning mechanism to improve the existing mixup methods on text classification tasks. se for mixup follows two stages: conducting data division based on the degree of difficulty and mixup based on the order from easy to hard. se can be used in various mixup methods to generate more adaptive and model-friendly pseudo samples for model training. also, to avoid over-confidence in the model, we propose a novel instance-specific label smoothing approach. extensive experiments on four popular mixup methods, embedmix, tmix, ssmix, and treemix, verify the effectiveness of our method. quantitative analyses and in-depth discussions show our method improves the generalization, and robustness of models.