Prototype-Guided Pseudo Labeling for Semi-Supervised Text Classification (2023)

Abstract: Semi-supervised text classification (SSTC) aims at text classification with few labeled data and massive unlabeled data. Recent works achieve this task by pseudo-labeling methods, with the belief that the unlabeled and labeled data have identical data distribution, and assign the unlabeled data with pseudo-labels as additional supervision. However, existing pseudo-labeling methods usually suffer from ambiguous categorical boundary issues when training the pseudo-labeling phase, and simply select pseudo-labels without considering the unbalanced categorical distribution of the unlabeled data, making it difficult to generate reliable pseudo-labels for each category. We propose a novel semi-supervised framework, namely ProtoS2, with prototypical cluster separation (PCS) and prototypical-center data selection (CDS) technology to address the issue. Particularly, PCS exploits categorical prototypes to assimilate instance representations within the same category, thus emphasizing low-density separation for the pseudo-labeled data to alleviate ambiguous boundaries. Besides, CDS selects central pseudo-labeled data considering the categorical distribution, avoiding the model from biasing on dominant categories. Empirical studies and extensive analysis with four benchmarks demonstrate the effectiveness of the proposed model.

## introduction traditional text classification has achieved profound success by leveraging numerous labeled data. however, acquiring large-scale labeled data is expensive in many real-world scenarios, making training effective classifiers using such approaches difficult. therefore, semi-supervised text classification (sstc), aiming at text classification requiring a few labeled data and massive unlabeled data, has become a new appealing technique. as a promising semi-supervised paradigm, the pseudo-labeling method @xcite @xcite ; li et al., 2022), which assigns pseudo-labels to unlabeled data instances as additional supervision, has drawn attention. they follow the well-known lowdensity separation assumption @xcite to generate pseudo-labels for the unlabeled data scattered in the high-density feature space, helping to identify the boundaries between classes. trained with the mixture of labeled and pseudolabeled data, the model perceives more abundant supervision signals and reduces empirical errors. existing models attempt to assign soft pseudolabels for unlabeled texts @xcite or keep pseudo-labels with lower uncertainty @xcite to alleviate the influence brought by the incorrect pseudo-labeled data. in practice, the underfitting of decision boundaries is usually detrimental to the model learning (shown in figure 1(a) ). more careful considerations are expected in properly utilizing these pseudo-labeled data near decision boundaries. in addition, the pseudo-labeling process may meet with bias from imbalanced data and will be biased toward dominant class data (shown in figure 1(b) ). this problem may further lead the dominant classes to dominate the follow-up training and assign more incorrect labels to the unlabeled data. believing that addressing both the underfitting of decision boundaries and the bias from the im-balanced data is crucial, we propose a prototypeguided pseudo labeling (pgpl) model to tackle the two problems simultaneously. pgpl improves the sstc performance with a prototype-anchored contrasting (pac) module and a prototype-guided pseudo labeling (pgp) module. specifically, in the pac module, we leverage the prototypical learning @xcite to construct prototypes with representations of labeled texts and classify unseen texts according to their distance from the prototype of each class. in this way, we force the text representations to locate in high-density areas, making the unreliable pseudo-labeled data near the boundary away from decision boundaries to address the problem, as demonstrated in figure 1 @xcite . furthermore, to alleviate the bias from the imbalanced data, our pgp module chooses reliable pseudo-labeled data by reducing the probability of assigning labels of dominant class for unlabeled data, and thus prevents the model from biasing on dominant classes, as demonstrated in figure 1(b) . finally, we conduct experiments on four commonly-used datasets, and the results demonstrate that our proposed pgpl model outperforms the state-of-the-art performance in various semisupervised settings. the experimental analysis also suggests that our pgpl model successfully separates the different classes and assigns reliable pseudo-labels to the unlabeled instances. these results confirm the effectiveness of pgpl. to summarize, our main contributions are in the following four folds: ## problem formulation in semi-supervised text classification, we are given a small set of labeled text data and a large set of un-labeled text data. formally, let c be the set of all labels of interest. ## methodology we propose a semi-supervised text classification model pgpl to tackle problems of underfitting near the decision boundary and bias from the imbalanced data. in this section, we first illustrate the overall model structure and introduce the fundamental sstc structure with supervised and unsupervised losses, then describe the prototypeanchored contrasting module and finally explain the prototype-guided pseudo labeling module.