Text Smoothing: Enhance Various Data Augmentation Methods on Text Classification Tasks (2022)

Abstract: Before entering the neural network, a token needs to be converted to its one-hot representation, which is a discrete distribution of the vocabulary. Smoothed representation is the probability of candidate tokens obtained from the pre-trained masked language model, which can be seen as a more informative augmented substitution to the one-hot representation. We propose an efficient data augmentation method, dub as text smoothing, by converting a sentence from its one-hot representation to controllable smoothed representation. We evaluate text smoothing on different datasets in a low-resource regime. Experimental results show that text smoothing outperforms various mainstream data augmentation methods by a substantial margin. Moreover, text smoothing can be combined with these data augmentation methods to achieve better performance.

## introduction data augmentation is a widely used technique, especially in the low-resource regime. it increases the size of the training data to alleviate overfitting and improve the robustness of deep neural networks. in the field of natural language processing (nlp), various data augmentation techniques have been proposed. one most commonly used method is to randomly select tokens in a sentence and replace them with semantically similar tokens to synthesize a new sentence @xcite . @xcite proposes contextual augmentation to predict the probability distribution of replacement tokens by using the lstm language model and sampling the replacement tokens according to the probability distribution. @xcite ) uses bert's @xcite masked language modeling (mlm) ## related work various nlp data augmentation techniques have been proposed and they are mainly divided into two categories: one is to modify raw input directly, and the other interferes with the embedding @xcite . the most commonly used method to modify the raw input is the token replacement: randomly select tokens in a sentence and replace them with semantically similar tokens to synthesize a new sentence. @xcite directly uses the synonym table wordnet @xcite for replacement. @xcite proposes contextual augmentation to predict the probability distribution of replacement tokens with two causal language models. @xcite ) extends contextual augmentation with bert's masked language modeling (mlm) to consider bi-directional context. @xcite softly augments a randomly chosen token in a sentence by replacing its one-hot representation with the distribution of the vocabulary provided by the causal language model in machine translation. unlike @xcite , we use mlm to generate smoothed representation, which considers the deep bi-directional context more adequately. and our method has better parallelism, which can efficiently obtain the smoothed representation of the entire sentence in one forward process. moreover, we propose to constrain smoothed representation more controllable through interpolation for classification tasks. ## conclusoins this article proposes text smoothing, an effective data augmentation method, by converting sentences from their one-hot representations to smoothing representations. in the case of a low data regime, text smoothing is significantly better than various data augmentation methods. furthermore, text smoothing can further be combined with various data augmentation methods to obtain better performance.