Effective Data Augmentation for Sentence Classification Using One VAE per Class (2022)

Abstract: In recent years, data augmentation has become an important field of machine learning. While images can use simple techniques such as cropping or rotating, textual data augmentation needs more complex manipulations to ensure that the generated examples are useful. Variational auto-encoders (VAE) and its conditional variant the Conditional-VAE (CVAE) are often used to generate new textual data, both relying on a good enough training of the generator so that it doesn’t create examples of the wrong class. In this paper, we explore a simpler way to use VAE for data augmentation: the training of one VAE per class. We show on several dataset sizes, as well as on four different binary classification tasks, that it systematically outperforms other generative data augmentation techniques.

## introduction data augmentation (da) has been shown to be an efficient technique for deep neural networks @xcite , consisting in artificially creating new labelled examples and therefore inflating the size of the dataset. while a great number of data augmentation techniques have been developed in recent years, most of these require external data, either in a database form @xcite , pretrained embeddings @xcite , or neural networks @xcite . even if these techniques help improve classifiers, they cannot be used in a variety of contexts, most notably in artificial intelligence for rare languages, where there simply isn't enough training data to pre-train efficient models @xcite , or where the collection of labelled data for fine-tuning is difficult. the ethical importance of considering smaller communities and other languages in machine learning research has been noted repeatedly @xcite , and consequently, it is important to develop techniques that can work for languages other than english. in this paper, we take a special look at generative models for data augmentation, and more specifically at variational auto-encoders (vae), for binary classification tasks. generative algorithms such as the vae are especially interesting because they do not require external data and, therefore, can be used on a variety of domains. in particular, we show that using one separate vae per class and generating new data by random sampling of the latent space is a very efficient way of inflating the size of a dataset. to the best of our knowledge, this technique has been considered only once before, in the context of balancing unbalanced datasets, and with somewhat disappointing results @xcite . contrary to them, we study the technique in the context of pure data augmentation, on more datasets and various starting sizes of datasets, and conclude that it is an efficient technique. we test two variations of this method, with or without sharing parameters between the vaes, and show that both variations are equally efficient. we compare this technique to two others generative techniques that are commonly used in the da literature, mainly using vaes as a paraphrasing system, or using a conditional vae (cvae) for directly generating from a given class. we show that they perform consistently worse than the separate vae approach. we surmise that this is because, while we lose the ability to perform style transfer between classes, we gain a stronger guarantee that the generated examples will not be of an erroneous class. we furthermore compare to another technique that showed excellent results, cbert @xcite , and show using one vae per class for da outperforms it as well. this paper is organized as follows: sections 2, 3, and 4 present respectively a brief summary of vaes, the relevant literature to our paper, and the description of our various generative da methods. then, section 5 and 6 present our datasets and the results of our experiments. finally, we present a discussion of the results in section 7, the broader implications of our work in section 8, and conclude in section 9. ## variational auto encoders vaes are generative models introducing a latent variable z ∼ p(z), where data points (in our case, sentences) are assumed to be generated following p(x|z). vaes follow the general auto-encoder structure, with an encoder that predicts the parameters associated with the latent distribution, most often a diagonal gaussian, and a decoder that takes samples from the latent distribution and transforms them into sentences. at training time, sampling is done using the reparametrization technique (kingma and welling, 2014), which samples from @xmath0, where ϵ ∼ n (0, 1). this allows the gradient to flow through µ and σ. the training objective is the elbo, or evidence lower bound: in order to use the vae for text, the standard solution is to replace both the encoder and the decoder with recurrent neural networks (rnn). we show a simple example of a textual vae model in figure 1 . this, however, often comes with the problem of kl-collapse, where the vae relies entirely on the powerful decoder to generate sentences and collapses the latent distribution to a single point. to prevent this, we resort to two common techniques, namely kl-annealing, which slowly brings the strength of the kl-term in the elbo from 0 to 1, and word dropout, which randomly masks words for the decoder while training @xcite . both of these techniques encourage the vae to rely on the latent distribution. ## related work data augmentation has been studied extensively @xcite . we focus our review on studies targeting data augmentation for sentence classification, as well as works using the vae for da, which are the most pertinent studies for our work. the general principle of data augmentation is fairly direct: starting with a dataset of fixed size, we generate new (hopefully) diverse data that we can then use to feed the classifier, diminishing generalization error. while da for images can perform simple operations, such as cropping or rotating the images @xcite , things are a little more complicated for textual data, since operations on words (such as replacing or deleting words), have a chance of generating examples of the wrong class. as such, a lot of the focus is put on class-coherence, which simply means that generated examples have the class we want them to. many approaches have been developed for data augmentation for sentence classification, the simplest ones consisting in performing word-level operations. for example, the authors of (marivate and sefara, 2020) use w2v @xcite for replacing random words in a sentence, picking words that are close to the original one in a pretrained embedding space. in @xcite , replacements are made using the part of speech information to ensure they only replace content words. more operations can also be considered, as in eda @xcite , which performs four operations on the sentences: replacement by a synonym, deletion of words, swapping words, and inserting synonyms of words at random positions. in eda, synonyms are found by using wordnet @xcite instead of pretrained embeddings. substitutions can also be made using pretrained neural networks with a masked word task, such as an lstm @xcite , or conditional-bert (cbert) @xcite , which is a simple extension of bert where we prepend the class of the example to the sentence, in the hope that it will learn to generate class-coherent outputs. generative models, and most notably vaes, are frequently used in data augmentation, often using a conditional model to directly generate from a given class @xcite @xcite . this, however, relies on the assumption that the cvae is learning to correctly generate from the wanted class. while cvaes have made tremendous progress in recent years, it remains a difficult task for textual cvae. the use of unconditional vaes is also commonly explored. in @xcite , the boundary decision is found in the latent space so that generation can be made on either side of it, depending on the class we want to generate data for. a popular technique is to use the vae as a paraphrase machine @xcite , encoding and decoding the data points and relying on the stochasticity of the system to generate variations of the examples. this technique, however, is also based on the hope that the algorithm will learn to generate from the same class as the input example. finally, vaes have also been used to generate new data that is not class dependent, such as spoken utterances for spoken language understanding @xcite . @xcite attempt, similarly to our paper, to use one vae per class for balancing data. they compare it to cvae as well as eda. they show, however, that generative algorithms barely outperform sampling strategies when balancing the data, on two datasets. contrary to them, we study the use of one vae per class in a pure data augmentation context, with already balanced classes, and show that it does work better than other algorithms. we also test all the algorithms on several dataset sizes and on four datasets, providing analysis and examples of why and how it works. last but not least, we show that it works with modern classification algorithms (in this case, bert), and show that it works better than state-of-the-art data augmentation. ## vae for data augmentation in this paper, we revisit the technique of @xcite , which trains one vae per class for generating new examples for unbalanced data, but use it for general data augmentation. we test two variants of this, one with parameter sharing (sharing of the en-coder and embeddings), denoted vae-linked, and one where everything is separated, denoted vae-sep. concretely, in vae-sep we initialize and train m vaes, where m is the number of classes, and each vae is trained only on the data of one class, that is to say on all examples x i |y i ϵc k , where y i is the class of the example x i , and c k is the class number k. for vae-linked, we initialize one vae model with m decoders, each decoding only examples of one class. when generating, we first choose the class we want to generate from, select the corresponding vae, and generate by randomly sampling from its latent space. this process is illustrated in figure 2 . figure 2 : illustration of the generation process of a negative movie review with the three vae models tested in this paper. in vae-par, the new examples are created by passing sentences through the vae and relying on its stochasticity to create paraphrases. in cvae, the sentence is generated by sampling from the latent space and conditioning the decoding on the negative class. finally, in vae-sep, we train two different vaes, one for the negative sentences and one for the positive ones. we generate negative examples by sampling from the latent space and using the negative decoder. in vae-linked, the generation happens identically to vae-sep, but while training, we share the encoder and the embeddings between classes. positive examples are generated in a similar way but passing a positive sentence, the positive label, or using the positive decoder. we compare this to two popular da techniques using vaes: conditional generation (cvae) and paraphrasing (vae-par). while both vae-sep and vae-linked lose the advantage of style transfer between classes and also have fewer data points to learn to generate sentences, it also greatly reduces the risk of generating examples of the wrong class. as mentioned, generative models for textual da are interesting due to the fact that they do not need external data to work. as such, we do not use pretrained embeddings in the rnns of the vaes 1 . we discuss in section 9 ideas for future research in a context where the use of external data is encouraged. ## datasets we compare the models on four binary sentence classification tasks. we use movie critics classification with sst-2 @xcite , detection of fake news articles with the fakenews dataset 2 , detection of ironic tweets with the dataset irony @xcite , and detection of subjective questions with subj @xcite . characteristics of the datasets are summarized in ## experiments because we are interested in domains where data is limited, we sample portions of the datasets to act as our initial training sets. we test 4 different datasets sizes: 500, 1000, 1500, and 2000. when sampling, we make sure to balance classes. we run each experiment 15 times and report the average results. as our baseline, we train bert without data augmentation. 3 we report in it is to note that this might not be ideal, as it also means that the smaller dataset sizes will have a stronger kl term from the beginning, but we leave the full fine-tuning for future work. in the same vein, variational auto-encoders can be finicky to train correctly, so while the hyperparameters presented in in addition to the four generative da algorithms mentioned in section 4, we compare ourselves to cbert foot_2 , a data augmentation algorithm that showed good results on sentence classification @xcite . cbert predicts masked words in a sentence using bert @xcite . to ensure class coherence, the label is prepended to the sentence so that bert learns to generate words of the right class. finally, we use bert from the jiant toolkit @xcite as our classifier. we also notice that cbert and cvae perform the second-best on average. while cbert performs well when the dataset size is bigger, it underperforms on small datasets. cvae on the other hand performs well on small dataset sizes, but badly on larger ones, similarly to the vae-par. finally, we observe that the larger the initial dataset size, the harder it is to get an augmentation of performance, as noticed in (dai and adel, 2020). 6 in figure 4 , we show the performances of our algorithms on the individual datasets. we see that while vae-sep globally outperforms other algorithms, it sometimes has more difficulties, such as for the irony or fakenews datasets. it is possible that this is due to the nature of the data itself, where differences between the classes come more from syntactic differences than from differences in vocabulary. this implies that chances of generat-500 1000 1500 2000 aver. baseline 81.2 83.8 85.6 86.8 84.4 cbert 82.2 84.5 85.8 86.9 84.9 vae-s 83.1 85.0 86.0 87.0 85.2 vae-l 83.0 84.8 85.9 87.0 85.2 vae-p 82.2 84.0 85.3 86.3 84.5 cvae 83.0 84.6 85.6 86.5 84.9 in ## analysis we showed that the vae-sep method outperformed other methods by 0.3% globally. in this section, we analyze the algorithms and posit some hypothesis as to why it works better. ## implications in this work, we show that vae-sep, a data augmentation algorithm that doesn't require external data, obtains a satisfying performance, even surpassing cbert, a data augmentation algorithm using a pretrained neural network. while the vae-sep algorithm, at the moment, only on bert for english sentence classification, the next step of the research is to test it on rare languages. a large portion of nlp research focuses on english text, and even as pretrained multilingual models are allowing a good performance on multilingual data, such as xlm (conneau and lample, 2019), mbart @xcite , or m2m100 @xcite , labelled data in the target language is still needed for fine-tuning. the vae-sep approach that we use in this paper does not require any form of external data, and is therefore usable for data augmentation for rare languages. we have also shown that it works well on bert, so we expect that it would also help on multilingual pretrained transformers, most directly mbert @xcite , but we leave the confirmation of this hypothesis for future work. ## conclusion and future work efficiently augmenting datasets is a central issue in machine learning. it can increase performance for all kinds of tasks, and has also been shown to help while distilling models @xcite . in this paper, we consider data augmentation using simple generative models based on the vae architecture, which has the advantage that they do not need external data to work. we compare two methods used in the literature, paraphrasing and conditional generation, to another where we train one separate vae per class. furthermore, we compare ourselves to an established method, cbert, which performs substitutions of random words in the sentence using a pre-trained conditionnal bert model. we show that vae-sep and vae-linked consistently outperform other methods (cbert, cvae, vae-par), with an average gain in accuracy of 0.3%, on four binary classification tasks and four initial dataset sizes. this study opens the door to interesting followup research. first, we contrasted generative approaches with cbert, which showed good performance. however, due to a lack of objective comparison in the literature, we cannot guarantee cbert delivers sota results. such a comparison would help the field hone in on efficient techniques and work with a common basis. it would also be interesting to observe how vae-sep algorithm performs on multi-class datasets, since each vae receives only members of one class and therefore would receive less training data on multiclass tasks. the vae-sep algorithm also has possibilities of improvements. while in this study we used basic textual vaes, there are many systems that have been developed for better vaes that could be used. as mentioned, we take interest in vaes because they allow data augmentation without using external data, which is useful for domains with limited data, but it would be interesting to see the performance if we allowed the usage of external data, for example by using pre-trained embeddings or even pre-trained transformers @xcite . finally, there is always the question of understanding why exactly data augmentation helps. while it is starting to be studied @xcite , we are far from fully understanding the da process. understanding it would not only help create stronger da algorithms for modern deep neural networks, but furthermore help understand how these networks learn data representation.