AEDA : An Easier Data Augmentation Technique for Text Classification (2021)

Abstract: This paper proposes AEDA (An Easier Data Augmentation) technique to help improve the performance on text classification tasks. AEDA includes only random insertion of punctuation marks into the original text. This is an easier technique to implement for data augmentation than EDA method (Wei and Zou, 2019) with which we compare our results. In addition, it keeps the order of the words while changing their positions in the sentence leading to a better generalized performance. Furthermore, the deletion operation in EDA can cause loss of information which, in turn, misleads the network, whereas AEDA preserves all the input information. Following the baseline, we perform experiments on five different datasets for text classification. We show that using the AEDA-augmented data for training, the models show superior performance compared to using the EDA-augmented data in all five datasets. The source code will be made available for further study and reproduction of the results.

## introduction text classification is a major area of study in natural language processing (nlp) with numerous applications such as sentiment analysis, toxicity detection, and question answering, to name but a few. in order to build text classifiers that perform well, the training data need to be large enough so that the model can generalize to the unseen data. however, for many machine learning (ml) applications and domains, there do not exist sufficient labeled data for training. in this situation, data augmentation (da) can provide a solution and help improve the performance of ml systems @xcite @xcite . da can be carried out in many different ways such as by modifying elements of the input sequence, namely word substitution, deletion, and insertion (wei and zou, figure 1 : average performance of the generated data using our proposed augmentation method (aeda) compared with that of the original and eda-generated data on five text classification tasks. using both eda and aeda, we added 9 augmented sentences to the original training set to train the models. for each task, we ran the models with 5 different seed numbers and took the average score. using a deep language model to do da can be complicated, while word replacement techniques with the help of a word thesaurus, even though a simple method, risks information loss due to the operations such as deletion and substitution. these operations can even result in changing the label of the input sequence @xcite , thus misleading the network. to address these problems, we propose an extremely simple yet effective approach called aeda (an easier data augmentation) which includes only the insertion of various punctuation marks into the input sequence. aeda preserves all the input information and does not mislead the network since it keeps the word order intact while changing their positions in that the words are shifted to the right. our extensive experiments show that aeda helps the models avoid overfitting (figure 1 ). ## related work although the textual content is always increasing, data augmentation is still a highly active area of research since for machine learning applications, especially the new ones, the initial annotated data are usually small. as a result, researchers are constantly coming up with innovative ideas to create new data from the available content. some have experimented at the input sequence level performing operations on words. for example, to improve machine translation quality, @xcite utilize substitution of common words with rare ones, thus providing more context for the rare words, while @xcite use back-translation where automatically translated data along with the original human-translated data are employed to train a neural machine translation system. wang and yang (2015) replaces words with their synonyms for classifying tweets. similarly, andreas (2020) replace sentence fragments from common categories with each other in order to produce new sentences. others have opted for using pre-trained language models such as bert @xcite . @xcite utilizes contextual augmentation, replacing the words with the prediction of a bidirectional language model at a desired position in the sentence. @xcite and @xcite utilize reinforcement learning with a conditional language model which is carried out by attaching the correct label to the input sequence when training @xcite . working with transformer model @xcite , @xcite propose mix-transformer where two input sentences and their corresponding labels are linearly interpolated to create new samples. @xcite make use of data noising which can be considered similar to our work with the difference that they replace words choosing from the unigram frequency distribution or insert the underscore character as a placeholder, whereas we insert punctuation characters which usually occur in sentences. the related works mostly use some auxiliary data or a complicated language model to produce augmented data. conversely, our method is extremely simple to implement and does not need any extra data. in addition, it shows superior performance to eda in both simple models such as rnns and cnns and deep models such as bert. ## aeda augmentation in order to insert the punctuation marks, we randomly choose a number between 1 and one-third of the length of the sequence which indicates how many insertions will be carried out. the reason is that we want to ensure there is at least on inserted mark and at the same time we do not want to insert too many punctuation marks as too much noise might have a negative effect on the model, although this effect can be investigated in future work. then, positions in the sequence are also specified in random as many as the selected number in the previous step. in the end, for each chosen position, a punctuation mark is picked randomly from the six punctuation marks in {".", ";", "?", ":", "!", ","}. original a sad , superior human comedy played out on the back roads of life . aug 1 a sad , superior human comedy played out on the back roads ; of life ; . aug 2 a , sad . , superior human ; comedy . played . out on the back roads of life . aug 3 : a sad ; , superior ! human : comedy , played out ? on the back roads of life . ## experimental setup since we compare our proposed method with @xcite , we used the same codebase as theirs with no changes in the implementation of the models. we executed the code using a geforce rtx 2070 gpu with 8 gb of memory. ## discussion comparing the results that we have gained in our experiments with the ones reported in @xcite , we can see some discrepancy, especially in the impact of eda on improving the performance of the models. we speculate that the difference can be caused by the inconsistency in the training and test sets. although we obtained the datasets from the same references they have specified, some of them are not divided into train and test datasets ready to be used. as mentioned in section 4.1, we randomly divided them into train and test sets. in addition, some of them have different sizes which can produce different results. with that said, to conduct a fair evaluation, we kept the same setting for all comparisons in terms of the utilized library and source code, train and test sets, number of augmentations, number of runs, batch size, and learning rate. ## conclusion and future work we proposed an easy data augmentation technique for text classification tasks. extensive experiments on five different datasets showed that this extremely simple method which uses punctuation marks outperforms the eda technique which includes random deletion, insertion, and substitution of words, on all the utilized datasets. the future work will focus on exploiting the proposed method regarding which punctuation marks can have more impact, which ones to add or discard, and how many of them can be used to achieve a better performance. in addition, the question whether the punctuation marks should be inserted randomly or some positions are more effective will be investigated.