Augmenting NLP models using Latent Feature Interpolations (2020)

Abstract: Models with a large number of parameters are prone to over-fitting and often fail to capture the underlying input distribution. We introduce Emix, a data augmentation method that uses interpolations of word embeddings and hidden layer representations to construct virtual examples. We show that Emix shows significant improvements over previously used interpolation based regularizers and data augmentation techniques. We also demonstrate how our proposed method is more robust to sparsification. We highlight the merits of our proposed methodology by performing thorough quantitative and qualitative assessments.

## introduction in this paper, we deal with the problem of sentence classification. current state-of-the-art deep learning approaches for sentence classification have millions of parameters, thus requiring a large number of training examples, which may be time-consuming and expensive to obtain. we present a new data augmentation technique called emix to improve the performance of current text classification models. in general, it is difficult to come up with rules for language transformation similar to image transformations; hence universal data augmentation techniques have not thoroughly been explored in nlp yet @xcite . one such technique is mixup -it uses systematic transformations to make sure the model trains on samples from the vicinity distribution @xcite along with the original distribution of the training data. mixup is a data-agnostic data augmentation method that is proven to be effective in computer vision tasks @xcite . @xcite extends mixup to text classification using interpolations in the embedding space. in our proposed method, we eliminate the common mean vector from the embeddings. we then interpolate hidden representations at multiple layers, including the embedding layer. this mixing method is also simple and easy to implement and leads to a further improvement in performance. ## methodology the main idea of mixup @xcite is that given two labeled data points (x i , y i ) and (x j , y j ), where x i and x j are two random samples and y i and y j are the one-hot representation of the label. the algorithm creates virtual training samples by linear interpolation of input as well as labels: let g(., θ) denote the classification model used, where θ denotes the model parameters. assuming this model has m layers, we choose to mix the hidden representations at the m-th layer, @xmath0, m ]. mathematically the m-th layer is denoted as g m (., θ), hence the hidden representation of the m-th layer is h @xmath1, θ). the 0-th layer is considered as the embedding layer. hence, for two text samples x i and x j , h i 0 = w e x i , h j 0 = w e x j and the following hidden representations are as follows: these hidden representations at the m-th layer are mixed using equation 3. we denote this mixed representation as h m . mixup at the m-th layer is thus defined as follows: the continued forward pass after the mixed hidden representation has been generated is defined as follows: the layers chosen for mixup are denoted by s where @xmath2, s 2 , ...} where each s @xmath3, m ]. the layer m where mixup occurs is chosen randomly from s with equal probability given to each layer in s and sampled separately for each pair of examples that are mixed. ## experiments we investigated the effectiveness of emix with five benchmark sentence classification tasks as used by @xcite , for a fair comparison. the following datasets are used : trec @xcite , sst-1 @xcite , sst-2 (binary classification) and subj @xcite . we evaluate our proposed mixup using the popular cnn @xcite and bert @xcite ) for sentence classification model. we compare with three recent text augmentation methods including eda @xcite , wordmixup (linear interpolation is applied on the word embedding level) and senmixup (linear interpolation is conducted on the layer before the softmax layer) @xcite . these augmentation strategies rely less on additional text resources or domain knowledge. in our experiments, we follow the exact implementation and settings in @xcite , @xcite , and @xcite . specifically, we use filter sizes of 3, 4, and 5, each with 100 feature maps; dropout rate of 0.5 and l2 regularization of 0.2 for the baseline cnn model. we use the huggingface @xcite implementation of the bert base model with 12 transformer blocks, 12 attention heads, and 110 million parameters. we use the default learning rate of 0.00002, dropout rate as 0.1, and batch size 8 for all experiments with bert. all models were trained on nvidia tesla k80 gpu. ## observations observation 1: impact of emix on sentence classification observation 2: performance after sparsification: over parameterized networks have a considerably large memory and environmental footprint . recent works @xcite have shown that test accuracy similar to a large network can be achieved with only a fraction of it's parameters. we check the impact of mixup on models with fewer parameters obtained through pruning. various pruning techniques have been suggested to approximate weight's importance @xcite . we test our models with weight pruning, where we mask out weights with magnitude lesser than a calculated threshold value. we observe that emix is more robust to pruning at all levels of sparsity ( ## qualitative analysis : class separability we use the class-to-class (c2c) separability @xcite defined by estimating the spread of the with-in class samples (intraclass) compared to the between-class (interclass) ones. the separability (s) is defined as : the features learned using emix lead to higher class separability as compared to vanilla bert and forms clusters with lesser inter-class overlap than vanilla model. also, from figure 1 we can see the health and technology class embeddings are better seperated using emix which were previously highly overlapped in the vanilla model. ## conclusion we introduced emix, a novel interpolation based mixing strategy for text classification that uses linear interpolations of hidden states. we performed thorough quantitative assessments to demonstrate how emix outperforms strong augmentation baselines, both in terms of accuracy and robustness to weight pruning. qualitatively, we also elucidate how emix allows for better separation between classes.