M ix T ext: Linguistically-Informed Interpolation of Hidden Space for Semi-Supervised Text Classification (2020)

Abstract: This paper presents MixText, a semi-supervised learning method for text classification, which uses our newly designed data augmentation method called TMix. TMix creates a large amount of augmented training samples by interpolating text in hidden space. Moreover, we leverage recent advances in data augmentation to guess low-entropy labels for unlabeled data, hence making them as easy to use as labeled data. By mixing labeled, unlabeled and augmented data, MixText significantly outperformed current pre-trained and fined-tuned models and other state-of-the-art semi-supervised learning methods on several text classification benchmarks. The improvement is especially prominent when supervision is extremely limited. We have publicly released our code at https://github.com/GT-SALT/MixText .

## introduction in the era of deep learning, research has achieved extremely good performance in most supervised learning settings @xcite . however, when there is only limited labeled data, supervised deep learning models often suffer from over-fitting @xcite . this strong dependence on labeled data largely prevents neural network models from being applied to new settings or real-world situations due to the need of large amount of time, money, and expertise to obtain enough labeled data. as a result, semi-supervised learning has received much attention to utilize both labeled and unlabeled data for different learning tasks, as unlabeled data is always much easier and cheaper to collect @xcite . this work takes a closer look at semi-supervised text classification, one of the most fundamental tasks in language technology communities. prior research on semi-supervised text classification can be categorized into several classes: (1) utilizing variational auto encoders (vaes) to reconstruct the sentences and predicting sentence labels with latent variables learned from reconstruction such as @xcite @xcite ; (2) encouraging models to output confident predictions on unlabeled data for selftraining like @xcite @xcite ; (3) performing consistency training after adding adversarial noise @xcite or data augmentations @xcite ; (4) large scale pretraining with unlabeld data, then finetuning with labeled data @xcite . despite the huge success of those models, most prior work utilized labeled and unlabeled data separately in a way that no supervision can transit from labeled to unlabeled data or from unlabeled to labeled data. as a result, most semisupervised models can easily still overfit on the very limited labeled data, despite unlabeled data is abundant. to overcome the limitations, in this work, we introduce a new data augmentation method, called tmix (section 3), inspired by the recent success of mixup @xcite on image classifications. tmix, as shown in figure 1 , takes in two text instances, and interpolates them in their corresponding hidden space. since the combination is continuous, tmix has the potential to create infinite mount of new augmented data samples, thus can drastically avoid overfitting. based on tmix, we then introduce a new semi-supervised learning method for text classification called mixtext (section 4) to explicitly model the relationships between labeled and unlabeled samples, thus overcoming the limitations of previous semi-supervised models stated above. in a nutshell, mixtext first guesses low-entropy labels for unlabeled data, then uses tmix to interpolate the label and unlabeled data. mixtext can facilitate mining implicit relations between sentences by encouraging models to behave linearly in-between training examples, and utilize information from unlabeled sentences while learning on labeled sentences. in the meanwhile, mixtext exploits several semi-supervised learning techniques to further utilize unlabeled data including selftarget-prediction @xcite , entropy minimization @xcite , and consistency regularization @xcite after back translations. to demonstrate the effectiveness of our method, we conducted experiments (section 5) on four benchmark text classification datasets and compared our method with previous state-of-the-art semi-supervised method, including those built upon models pre-trained with large amount of unlabeled data, in terms of accuracy on test sets. we further performed ablation studies to demonstrate each component's influence on models' final performance. results show that our mixtext method significantly outperforms baselines especially when the given labeled training data is extremely limited. ## tmix in this section, we extend mixup-a data augmentation method originally proposed by @xcite for images-to text modeling. the main idea of mixup is very simple: given two labeled data points (x i , y i ) and (x j , y j ), where x can be an image and y is the one-hot representation of the label, the algorithm creates virtual training samples by linear interpolations: to this end, we propose a novel method to overcome this challenge -interpolation in textual hidden space. given a sentence, we often use a multilayer model like bert @xcite to encode the sentences to get the semantic representations, based on which final predictions are made. some prior work @xcite has shown that decoding from an interpolation of two hidden vectors generates a new sentence with mixed meaning of two original sentences. motivated by this, we propose to apply interpolations within hidden space as a data augment method for text. for an encoder with l layers, we choose to mixup the hidden representation at the m-th layer, @xmath0, l]. as demonstrated in figure 1 , we first compute the hidden representations of two text samples separately in the bottom layers. then we mix up the hidden representations at layer m, and feed the interpolated hidden representations to the upper layers. mathematically, denote the l-th layer in the encoder network as g l (.; θ), hence the hidden representation of the l-th layer can be computed as h @xmath1; θ). for two text samples x i and x j , define the 0-th layer as the embedding layer, i.e., h i 0 = w e x i , h j 0 = w e x j , then the hidden representations of the two samples from the lower layers are: the mixup at the m-th layer and continuing forward passing to upper layers are defined as: we call the above method tmix and define the new mixup operation as the whole process to get hl : by using an encoder model g(.; θ), tmix interpolates textual semantic hidden representations as a type of data augmentation. in contrast with mixup defined in the data space in equation 1, tmix depends on an encoder function, hence defines a much broader scope for computing interpolations. for ease of notation, we drop the explicit dependence on g(.; θ), λ and m in notations and denote it simply as tmix(x i , x j ) in the following sections. in our experiments, we sample the mix parameter λ from a beta distribution for every batch to perform the interpolation : in which α is the hyper-parameter to control the distribution of λ. in tmix, we mix the labels in the same way as equation 2 and then use the pairs ( hl , ỹ) as inputs for downstream applications. instead of performing mixup at random input layers like @xcite , choosing which layer of the hidden representations to mixup is an interesting question to investigate. in our experiments, we use 12-layer bert-base @xcite as our encoder model. recent work @xcite has studied what bert learned at different layers. specifically, the authors found {3,4,5,6,7,9,12} layers have the most representation power in bert and each layer captures different types of information ranging from surface, syntactic to semantic level representation of text. for instance, the 9-th layer has predictive power in semantic tasks like checking random swapping of coordinated clausal conjuncts, while the 3-rd layer performs best in surface tasks like predicting sentence length. building on those findings, we choose the layers that contain both syntactic and semantic information as our mixing layers, namely @xmath2, 9, 12}. for every batch, we randomly sample m, the layer to mixup representations, from the set m computing the interpolation. we also performed ablation study in section 5.5 to show how tmix's performance changes with different choice of mix layer sets. text classification note that tmix provides a general approach to augment text data, hence can be applied to any downstream tasks. in this paper, we focus on text classification and leave other applications as potential future work. in text classification, we minimize the kl-divergence between the mixed labels and the probability from the clas-sifier as the supervision loss: ## semi-supervised mixtext in this section, we demonstrate how to utilize the tmix to help semi-supervised learning. given a limited labeled text set x @xmath3, ..., x l n }, with their labels y @xmath4, ..., y l n } and a large unlabeled set x @xmath5, ..., x u m }, where n and m are the number of data points in each set. y l @xmath6, 1} c is a one-hot vector and c is the number of classes. our goal is to learn a classifier that efficiently utilizes both labeled data and unlabeled data. we propose a new text semi-supervised learning framework called mixtext foot_0 . the core idea behind our framework is to leverage tmix both on labeled and unlabeled data for semi-supervised learning. to fulfill this goal, we come up a label guessing method to generate labels for the unlabeled data in the training process. with the guessed labels, we can treat the unlabeled data as additional labeled data and perform tmix for training. moreover, we combine tmix with additional data augmentation techniques to generate large amount of augmented data, which is a key component that makes our algorithm work well in setting with extremely limited supervision. finally, we introduce an entropy minimization loss that encourages the model to assign sharp probabilities on unlabeled data samples, which further helps to boost performance when the number of classes c is large. the overall architecture is shown in figure 2 . we will explain each component in detail. ## conclusion to alleviate the dependencies of supervised models on labeled data, this work presented a simple but effective semi-supervised learning method, mix-text, for text classification, in which we also introduced tmix, an interpolation-based augmentation and regularization technique. through experiments on four benchmark text classification datasets, we demonstrated the effectiveness of our proposed tmix technique and the mixup model, which have better testing accuracy and more stable loss trend, compared with current pre-training and fine-tuning models and other state-of-the-art semi-supervised learning methods. for future direction, we plan to explore the effectiveness of mixtext in other nlp tasks such as sequential labeling tasks and other real-world scenarios with limited labeled data.