An Effective Deployment of Diffusion LM for Data Augmentation in Low-Resource Sentiment Classification (2024)

Abstract: Sentiment classification (SC) often suffers from low-resource challenges such as domain-specific contexts, imbalanced label distributions, and few-shot scenarios. The potential of the diffusion language model (LM) for textual data augmentation (DA) remains unexplored, moreover, textual DA methods struggle to balance the diversity and consistency of new samples. Most DA methods either perform logical modifications or rephrase less important tokens in the original sequence with the language model. In the context of SC, strong emotional tokens could act critically on the sentiment of the whole sequence. Therefore, contrary to rephrasing less important context, we propose DiffusionCLS to leverage a diffusion LM to capture in-domain knowledge and generate pseudo samples by reconstructing strong label-related tokens. This approach ensures a balance between consistency and diversity, avoiding the introduction of noise and augmenting crucial features of datasets. DiffusionCLS also comprises a Noise-Resistant Training objective to help the model generalize. Experiments demonstrate the effectiveness of our method in various low-resource scenarios including domain-specific and domain-general problems. Ablation studies confirm the effectiveness of our framework’s modules, and visualization studies highlight optimal deployment conditions, reinforcing our conclusions.

## introduction sentiment classification is a crucial application of text classification (tc) in natural language processing (nlp) and can play a crucial role in multiple areas. however, nlp applications in domainspecific scenarios, such as disasters and pandemics, often meet with low-resource conditions, especially domain-specific problems, imbalance data distribution, and data deficiency (sedinkina et al., 2022; † corresponding author. @xcite @xcite . recently, the birth of pre-trained language models (plms) and large language models (llms) have advanced the nlp field, giving birth to numerous downstream models based on them. on the one hand, these plms take the models to a new height of performance, on the other hand, since these models are highly data-hungry, they struggle to perform satisfactorily on most tasks under noisy, data-sparse and low-resource conditions @xcite @xcite . ## methodology sentiment classification models often overfit and lack generalization due to sample deficiency. to address this, we propose diffusioncls, consisting of label-aware noise schedule, label-aware prompting, conditional sample generation, and noise-resistant training. a diffusion lm-based sample generator is integrated to generate new samples from the original dataset, enhancing tc model performance. figure 1 illustrates diffusioncls. the diffusion lm-based sample generator generates new samples for data augmentation, while the tc model is trained for the specific task. label-aware prompting and label-aware noise schedule are crucial for training the sample generator, and conditional sample generation and noise-resistant training contribute to the training of the tc model. ## conclusion in this work, we propose diffusioncls, a novel approach tackling sc challenges under low-resource conditions, especially in domain-specific and uneven distribution scenarios. utilizing a diffusion lm, diffusioncls captures in-domain knowledge to generate high-quality pseudo samples maintaining both diversity and consistency. this method surpasses various kinds of data augmentation techniques. our experiments demonstrate that diffu-sioncls significantly enhances sc performance across various domain-specific and multilingual datasets. ablation and visualization studies further validate our approach, emphasizing the importance of balancing diversity and consistency in pseudo samples. diffusioncls presents a robust solution for data augmentation in low-resource nlp applications, paving a promising path for future research.