Neural Networks Against (and For) Self-Training: Classification with Small Labeled and Large Unlabeled Sets (2023)

Abstract: We propose a semi-supervised text classifier based on self-training using one positive and one negative property of neural networks. One of the weaknesses of self-training is the semantic drift problem, where noisy pseudo-labels accumulate over iterations and consequently the error rate soars. In order to tackle this challenge, we reshape the role of pseudo-labels and create a hierarchical order of information. In addition, a crucial step in self-training is to use the classifier confidence prediction to select the best candidate pseudo-labels. This step cannot be efficiently done by neural networks, because it is known that their output is poorly calibrated. To overcome this challenge, we propose a hybrid metric to replace the plain confidence measurement. Our metric takes into account the prediction uncertainty via a subsampling technique. We evaluate our model in a set of five standard benchmarks, and show that it significantly outperforms a set of ten diverse baseline models. Furthermore, we show that the improvement achieved by our model is additive to language model pretraining, which is a widely used technique for using unlabeled documents.

## introduction text classification has achieved tremendous success in the past decade thanks to the advancement in deep neural networks. even though the introduction of contextual word embeddings and language model pretraining @xcite has greatly reduced the reliance on large manually annotated datasets, the current over-parametrized models are still prone to overfitting. to further reduce this reliance one can use unlabeled data @xcite . in this article, we use the properties of neural networks and develop a self-training model, termed robust self-training (rst), for low-data regime text classification. self-training (also known as pseudo-labeling) is iterative @xcite , and in each iteration unlabeled documents are automatically annotated and augmented with labeled documents. previous studies @xcite report that self-training suffers from the semantic drift problem. that is, as the iterations are carried on, spurious pseudo-labels are generated and added to the labeled documents. this eventually distorts the class boundaries and drifts the original class centroids. to address this problem, inspired by the catastrophic forgetting phenomenon in neural networks @xcite , we propose a novel procedure to reshape the role of pseudo-labels in the algorithm. we also aim to overcome a weakness of neural networks in this algorithm. self-training relies on prediction confidence to select the best candidate documents. in this framework, the classifier output is interpreted as prediction confidence @xcite . self-training performance deteriorates in the settings that the used classifier is unable to accurately estimate the prediction confidence @xcite . neural networks suffer from such a problem, because their outputs are mis-calibrated @xcite . to address this problem, we propose a novel metric to replace the plain confidence measurement. our metric takes into account the prediction uncertainty via a subsampling algorithm. we use a set of five standard benchmarks to evaluate our model. the selected datasets cover a wide spectrum of documents, ranging from formal news documents to highly informal social media documents. we also compare our model with a set of ten methods, including approaches that use variants of self-training, use multiple classifiers, use multi-view learning, and use various uncertainty metrics. the experiments signify to the superiority of our model. additionally, we analyze our model and demonstrate that the improvement achieved by our model is additive to the performance of domain specific language model pretraining. the contributions of our work are as follows: 1) we mitigate the semantic drift problem in selftraining by reshaping the role of pseudo-labeled documents and creating a hierarchical order of information. 2) we enhance the pseudo-label selection in self-training by proposing a novel selection metric to replace the plain confidence measurement. our metric is particularly advantageous when neural networks are used as the underlying classifier, because these classifiers are overconfident in their predictions @xcite . 3) through an extensive set of experiments with five datasets and ten baselines we show that our model is highly resistant to noisy pseudo-labels, yields an additive improvement to domain specific language model pretraining, and outperforms the state of the art. ## related work neural networks in self-training. self-training or pseudo-labeling @xcite is a semi-supervised learning algorithm. previous studies investigate various aspects of this algorithm and aim for filling the niches. for instance, @xcite integrate mixup @xcite with the oversampling of labeled documents, amiri (2019) proposes a new document sampling strategy, @xcite and @xcite report that adding noise to pseudo-labels and the hidden layers of a network enhances the model performance-the latter for the sequence generation task, and @xcite contrast self-training and pretraining and conclude that under certain conditions the former outperforms the latter. @xcite propose a multi-view self-training model to incorporate domain-knowledge, @xcite propose a feedback loop across self-training iterations, @xcite propose a model to incorporate weakly supervised domain specific rules, @xcite report that pre-training a model with an auxiliary nli task enhances self-training, and @xcite reduces the variance of pseudolabels within each class using an angular loss. as opposed to our research, none of these studies propose a model to maintain a balance between the set of pseudo-labels and the set of manual labels. additionally, they don't analyze the deterioration of performance during the self-training iterations, and consequently have no defense against this fundamental weakness. uncertainty measurement in nlp. confidence in model prediction, is the amount of trust in the predicted class label compared to the other class labels @xcite . uncertainty in model prediction, is the amount of trust in the entire prediction regardless of the predicted label @xcite . the research on the efficacy of uncertainty in semi-supervised learning is scarce. @xcite propose to filter out uncertain predictions before the candidate selection step, @xcite apply a set of thresholds to filter out uncertain and unconfident predictions, and @xcite experiment with various uncertainty metrics and report that uncovering the heteroscedastic uncertainty (the intrinsic data uncertainty) @xcite is the best strategy on average. as opposed to our work, none of these studies propose an integrated metric for selecting pseudolabels. additionally, after selecting the pseudolabels, they don't propose any strategy to restrain the noisy labels from polluting the labeled set. ensemble and multi-view models. there exist models that use multiple classifiers, examples include variants of tri-training @xcite , variants of co-training @xcite @xcite , and other ad hoc ensemble models @xcite @xcite . as opposed to our work, these models rely only on the confidence of classifiers. no coherent uncertainty interpretation has been proposed for them. additionally, they use ensembling in the prediction stage, whereas, we employ only one classifier for this purpose, which is more resource efficient. semantic drift in self-training. semantic drift in self-training @xcite occurs when spurious pseudo-labels accumulate over time and distort the distribution of labeled documents. in the context of neural networks, the research in this area is sparse. one approach is to avoid pseudo-labels altogether, and use unlabeled documents differently @xcite @xcite . nonetheless, these alternative methods don't necessarily compete with self-training and can co-exist with it inside a framework. to address semantic drift directly, existing approaches mainly aim for explicitly adjusting pseudo-labels. @xcite use an angular loss function to project pseudo-labels. @xcite assume pseudo-labels evolve in a stochastic process and normalize their values. in terms of the role of pseudo-labels in self-training, our algorithm can be taken as the generalized form of the algorithm proposed by @xcite . ## proposed method in a typical self-training model @xcite , there is a set l of labeled data, and a set u of unlabeled data. a predictive model is trained on l and is used to probabilistically label u . then, given a hyper-parameter θ, as the minimum confidence threshold, the confidently labeled documents in u and their associated pseudo-labels are selected and added to l. this procedure is iterative. in this framework, there is no constraint on the choice of the underlying model, except that it is required to assign a confidence score to each pseudo-label. there are two challenges to face in this setting: 1) self-training suffers from the semantic drift problem @xcite . that is, as we increase the number of iterations, the error rate accumulates and the class boundaries are distorted. 2) neural networks are overconfident in their predictions @xcite , and as we discuss in section 3.2, this shortcoming deteriorates the quality of the pseudo-labels in each iteration. to address these challenges, we present robust self-training (rst ). algorithm 1 provides an overview of rst , with two classifiers, in struc-tured english. lines 12 to 18 demonstrate one iteration of the algorithm, which is repeated till the set of unlabeled documents u is exhausted. the iteration begins by initializing the classifiers c 1 and c 2 . then, it continues by sampling from the set of pseudo-labels s and distilling it @xcite into the classifier c 1 . foot_0 then, another sample from the set of labeled documents l is taken to further train c 1 using equation 1 (see section 3.1). these steps are re-taken to train the second classifier c 2 . finally, c 1 and c 2 are used in equation 2 (see section 3.2) to label and score the documents in the set u . the top documents are removed from u and are added to the set s. since we have multiple classifiers labeling each document (in this case two classifiers), we store the average of the outputs in s. on line 19, the entire set s is used to pretrain the final classifier c, and on line 20, the set l is used to finetune c using equation 1. in the following two sections we discuss how our algorithm can address the two aforementioned challenges. l : set of labeled documents 4: u : set of unlabeled documents 5: return: 6: trained classifier on l and u 7: execute: 8: set k to 100 // hyper-parameter (step size) 9: set r to 70 // hyper-parameter (sample ratio) 10: set s to empty // the set of pseudo-labels 11: initialize the classifiers c1 and c2 13: sample r% of s, order the data as described in section 3.1, and use it to train c1 14: sample r% of l and use in equation 1 for c1 15: sample r% of s, order the data as described in section 3.1, and use it to train c2 16: sample r% of l and use in equation 1 for c2 17: use c1 and c2 to label u and then score the documents using equation 2 18: remove the top k documents from u and add them to s 19: order the set s as described in section 3.1, and use it to train the classifier c 20: use the set l in equation 1 to further train c 21: ## experimental setup in the current and in the next sections we describe our experimental setup and our results. ## conclusions in this paper we proposed a semi-supervised text classifier. our model is based on the self-training paradigm and employs neural network properties to enhance the bootstrapping procedure. specifically, we use a subsampling technique to overcome the poor calibration of neural networks and to improve the candidate selection. then, we exploit the catastrophic forgetting phenomenon in neural networks to alleviate the semantic drift problem. we evaluated our model in five public datasets and showed that it outperforms ten baselines.