Semi-Supervised Text Classification with Balanced Deep Representation Distributions (2021)

Abstract: Semi-Supervised Text Classification (SSTC) mainly works under the spirit of self-training. They initialize the deep classifier by training over labeled texts; and then alternatively predict unlabeled texts as their pseudo-labels and train the deep classifier over the mixture of labeled and pseudo-labeled texts. Naturally, their performance is largely affected by the accuracy of pseudo-labels for unlabeled texts. Unfortunately, they often suffer from low accuracy because of the margin bias problem caused by the large difference between representation distributions of labels in SSTC. To alleviate this problem, we apply the angular margin loss, and perform Gaussian linear transformation to achieve balanced label angle variances, i.e., the variance of label angles of texts within the same label. More accuracy of predicted pseudo-labels can be achieved by constraining all label angle variances balanced, where they are estimated over both labeled and pseudo-labeled texts during self-training loops. With this insight, we propose a novel SSTC method, namely Semi-Supervised Text Classification with Balanced Deep representation Distributions (S2TC-BDD). To evaluate S2TC-BDD, we compare it against the state-of-the-art SSTC methods. Empirical results demonstrate the effectiveness of S2TC-BDD, especially when the labeled texts are scarce.

## introduction semi-supervised learning (ssl) refers to the paradigm of learning with labeled as well as unlabeled data to perform certain applications @xcite . especially, developing effective ssl models for classifying text data has long been a goal for the studies of natural language processing, because labeled texts are difficult to collect in many real-world scenarios. formally, this research topic is termed as semi-supervised text classification (sstc), which nowadays draws much attention from the community @xcite @xcite . to our knowledge, the most recent sstc methods mainly borrow ideas from the successful patterns of supervised deep learning, such as pretraining and fine-tuning @xcite @xcite @xcite . generally, those methods perform deep representation learning on unlabeled texts followed by supervised learning on labeled texts. however, a drawback is that they separately learn from the labeled and unlabeled texts, where, specifically, the deep representations are trained without using the labeling information, resulting in potentially less discriminative representations as well as worse performance. to avoid this problem, other sstc methods combine the traditional spirit of self-training with deep learning, which jointly learn the deep representation and classifier using both labeled and unlabeled texts in a unified framework @xcite @xcite @xcite . to be specific, this kind of methods initializes a deep classifier, e.g., bert @xcite with angular margin (am) loss @xcite , by training over labeled texts only; and then it alternatively predicts unlabeled texts as their pseudo-labels and trains the deep classifier over the mixture of labeled and pseudo-labeled texts. accordingly, both labeled and unlabeled texts can directly contribute to the deep classifier training. generally speaking, for deep self-training methods, one significant factor of performance is the accuracy of pseudo-labels for unlabeled texts. unfortunately, they often suffer from low accuracy, where one major reason is the margin bias problem. to interpret this problem, we look around the am loss with respect to the label angle, i.e., the angles between deep representations of texts and weight vectors of labels. for unlabeled texts, the pseudo-labels are predicted by only ranking the label angles, but neglecting the difference between label angle variances, i.e., the variance of label angles of texts within the same label, which might be much large in ssl as illustrated in fig. 1 . in this context, the boundary of am loss is actually not the optimal one, potentially resulting in lower accuracy for pseudo-labels (see fig. 2(a) ). to alleviate the aforementioned problem, we propose a novel sstc method built on bert with am loss, namely semi-supervised text classification with balanced deep representation distributions (s 2 tc-bdd). most specifically, in s 2 tc-bdd, we suppose that the label angles are drawn from each label-specific gaussian distribution. therefore, for each text we can apply linear transformation operations to balance the label angle variances. this is equivalent to moving the boundary to the optimal one, so as to eliminate the margin bias (see examples in fig. 2(b) ). we can estimate each label angle variance over both labeled and pseudo-labeled texts during the self-training loops. we evaluate the proposed s 2 tc-bdd method by comparing the most recent deep sstc methods. experimental results indicate the superior performance of s 2 tc-bdd even with very few labeled texts. ## related work the pre-training and fine-tuning framework has lately shown impressive effectiveness on a variety of tasks @xcite @xcite ; devlin @xcite @xcite @xcite @xcite . they mainly perform deep representation learning on generic data, followed by supervised learning for downstream tasks. several sstc methods are built on this framework @xcite @xcite @xcite . for instance, the variational methods for pretraining in resourcelimited environments (vampire) @xcite first pre-trains a variational auto-encoder (vae) model on unlabeled texts, and then trains a classifier on the augmentation representations of labeled texts computed by the pre-trained vae. however, the vae model is trained without using the labeling information, resulting in potentially less discriminative representations for labeled texts. recent works on sstc mainly focus on deep self-training @xcite @xcite @xcite , which can jointly learn deep representation and classifier using both labeled and unlabeled texts in a unified framework. it is implemented by performing an alternative process, in which the pseudo-labels of unlabeled texts are updated by the current deep classifier, and then the deep classifier is retrained over both labeled and pseudo-labeled texts. for example, the virtual adversarial training (vat) method @xcite follows the philosophy of making the classifier robust against random and local perturbation. it first generates the predictions of original texts with the current deep classifier and then trains the deep classifier by utilizing a consistency loss between the original predictions and the outputs of deep classifier over noise texts by applying local perturbations to the embeddings of original texts. further, the work in @xcite combines maximum likelihood, adversarial training, virtual adversarial training, and entropy minimization in a unified objective. furthermore, rather than applying local perturbations, unsupervised data augmentation (uda) @xcite employs consistency loss between the predictions of unlabeled texts and corresponding augmented texts by data augmentation techniques such as back translations and tf-idf word replacements. the work @xcite exploits cross-view training by matching the predictions of auxiliary prediction modules over the restricted views of unlabeled texts (e.g., only part of sentence) with ones of primary prediction module over the corresponding full views. orthogonal to the aforementioned self-training sstc methods, our s 2 tc-bdd further considers the margin bias problem by balancing the label angle variances. this is beneficial for more accurate pseudo-labels for unlabeled texts, so as to boost the performance of sstc tasks. ## the proposed s 2 tc-bdd method in this section, we describe the proposed deep selftraining sstc method, namely semi-supervised text classification with balanced deep representation distributions (s 2 tc-bdd). formulation of sstc consider a training dataset d consisting of a limited labeled text set and a large unlabeled text set d @xmath0. specifically, let x l i and x u j denote the word sequences of labeled and unlabeled texts, respectively; and let y l @xmath1, 1} k denote the corresponding one-hot label vector of x l i , where if the text is associated with the k-th label, or y l @xmath2. we declare that n l , n u , and k denote the numbers of labeled texts, unlabeled texts and category labels, respectively. in this paper, we focus on the paradigm of inductive sstc, whose goal is to learn a classifier from the training dataset d with both labeled and unlabeled texts. the important notations are described in ## conclusion in this paper, we propose a novel self-training sstc method, namely s 2 tc-bdd. our s 2 tc-bdd addresses the margin bias problem in sstc by balancing the label angle variances, i.e., the variance of label angles of texts within the same label. we estimate the label angle variances with both labeled and unlabeled texts during the self-training loops. to constrain the label angle variances to be balanced, we design several gaussian linear transformations and incorporate them into a well established am loss. our s 2 tc-bdd empirically outperforms the existing sstc baseline methods.