On-the-fly Denoising for Data Augmentation in Natural Language Understanding (2024)

Abstract: Data Augmentation (DA) is frequently used to provide additional training data without extra human annotation automatically.However, data augmentation may introduce noisy data that impairs training.To guarantee the quality of augmented data,existing methods either assume no noise exists in the augmented data and adopt consistency training or use simple heuristics such as training loss and diversity constraints to filter out “noisy” data.However, those filtered examples may still contain useful information, and dropping them completely causes a loss of supervision signals.In this paper, based on the assumption that the original dataset is cleaner than the augmented data, we propose an on-the-fly denoising technique for data augmentation that learns from soft augmented labels provided by an organic teacher model trained on the cleaner original data.To further prevent overfitting on noisy labels, a simple self-regularization module is applied to force the model prediction to be consistent across two distinct dropouts.Our method can be applied to general augmentation techniques and consistently improve the performance on both text classification and question-answering tasks.

## introduction the development of natural language understanding (nlu) comes along with the efforts in curating large-scale human-annotated datasets @xcite . the performance of nlp models usually highly correlates with the quantity and quality of training data. however, human data annotations are usually expensive to acquire and hard to scale @xcite . to address this challenge, automatic data augmentation becomes an attractive approach to effectively ## method this section introduces the problem formulation ( §3.1) and our odda framework ( §3.2- §3.3). ## experiments this section introduces experimental settings and results analysis. we evaluate on two representative tasks in nlu, few-shot text classification (section §4.1) and multiple-choice (commonsense) question answering (section §4.2). we use eda @xcite as a representative tokenlevel based augmentation method for text classification, and use generative data augmentation (g-daug) @xcite to explore task-aware sentence-level augmentation methods for hard qa tasks that require commonsense reasoning abilities. in section §4.3, we provide ablation studies to show the effect of odda under synthetic noise on augmented data, the influence of hyperparameters, and the effect of denoising modules. ## conclusion in this paper, we address the problem of improving data augmentation via denoising, and propose an efficient on-the-fly data augmentation denoising framework that leverages a teacher model trained on the cleaner original dataset for soft label correction and a self-regularized denoising loss for general denoising. such a denoising pipeline can well benefit the tasks with limited annotated data and noisy augmented data. experiments show that our denoising framework performs consistently better than the baselines of filtering, re-weighting, and consistency training, with both token-level and sentence-level data augmentation methods on fewshot text classification and commonsense questionanswering tasks.