J oint M atch: A Unified Approach for Diverse and Collaborative Pseudo-Labeling to Semi-Supervised Text Classification (2023)

Abstract: Semi-supervised text classification (SSTC) has gained increasing attention due to its ability to leverage unlabeled data. However, existing approaches based on pseudo-labeling suffer from the issues of pseudo-label bias and error accumulation. In this paper, we propose JointMatch, a holistic approach for SSTC that addresses these challenges by unifying ideas from recent semi-supervised learning and the task of learning with noise. JointMatch adaptively adjusts classwise thresholds based on the learning status of different classes to mitigate model bias towards current easy classes. Additionally, JointMatch alleviates error accumulation by utilizing two differently initialized networks to teach each other in a cross-labeling manner. To maintain divergence between the two networks for mutual learning, we introduce a strategy that weighs more disagreement data while also allowing the utilization of high-quality agreement data for training. Experimental results on benchmark datasets demonstrate the superior performance of JointMatch, achieving a significant 5.13% improvement on average. Notably, JointMatch delivers impressive results even in the extremely-scarce-label setting, obtaining 86% accuracy on AG News with only 5 labels per class. We make our code available at https://github.com/HenryPengZou/JointMatch.

## introduction the success of deep learning models often heavily depends on the availability of large amounts of labeled data @xcite . however, the labeled data for many tasks are often expensive, difficult, and time-consuming to obtain. by contrast, acquiring unlabeled data is more costeffective and convenient in many scenarios. this has led to a surge of interest in semi-supervised learning, which aims to enhance learning performance with limited labeled samples by leveraging large amounts of unlabeled data @xcite . recently, the combination of pseudo-labeling and consistency regularization has become a popular paradigm for semi-supervised learning @xcite @xcite . pseudo-labeling @xcite uses a fixed threshold to select the model's highconfidence predictions as pseudo-labels for further training, whereas consistency regularization @xcite enforces the model to make similar predictions for perturbed versions of the same data. for example, uda @xcite applies strong data augmentations, such as back-translation, to unlabeled data and minimizes the divergence between model predictions of input and its augmented views. fixmatch @xcite uses the pseudo-label generated from weakly-augmented unlabeled data to supervise the strongly augmented version of the same data. sat @xcite improves fix-match by training a meta-learner to re-rank different augmentations based on their similarities with the original data. these methods require a pre-defined high-confidence threshold to generate high-quality pseudo-labels for good performance. however, there are some potential limitations: (1) setting a fixed threshold for pseudo-label selection neglects the varied difficulties of learning different classes @xcite . this can cause the model bias toward easy classes, as more pseudo-labels will be generated for current easy classes (see figure 1a ); and (2) if these pseudo-labels are incorrect and used to train the model, the model can be worse and produce more inaccurate pseudo-labels, progressively accumulating its error and degenerating its performance (see figure 1c ) @xcite . to address these issues, we propose joint-match, a diverse and collaborative pseudo-labeling approach for semi-supervised text classification (sstc). jointmatch is a holistic framework that unifies ideas from recent semi-supervised learning and the task of learning with noise for sstc. inspired by flexmatch @xcite and freematch @xcite , we adaptively adjust classwise thresholds based on the estimated learning status for different classes at different times. this enables difficult classes at the current iteration to have lower local thresholds, thereby facilitating more pseudo-labels to be produced and used for learning these classes (see figure 1b ). following the idea of co-training @xcite , jointmatch simultaneously trains two differently initialized networks and uses them to teach each other in a cross-labeling manner to alleviate error accumulation, as different networks can filter different types of noise. nevertheless, with the increase of training iterations, those networks will slowly converge and reduce to one network and thus will again suffer from the issue of error accumulation. inspired by co-teaching+ @xcite and decoupling @xcite , we propose to give more weight to disagreement data to keep the two networks diverged while also allowing the utilization of high-quality agreement data for training. the relationship and difference between our jointmatch and related techniques are discussed in detail in section 2.5. we evaluate the proposed jointmatch on three commonly studied sstc benchmark datasets. experimental results indicate the superior performance of jointmatch, obtaining a significant 5.13% average improvement over the latest work sat. we also analyze the performance of jointmatch with varying numbers of labeled data. the results show that jointmatch can deliver impressive results even in the extremely-scarce-labels setting, achieving 86% accuracy on ag news with only 5 labels per class. we provide comprehensive ablation studies and analysis to understand each part of jointmatch. the main pipeline for jointmatch is shown in figure 2 . for each batch of unlabeled data u, we first apply both weak data augmentation α(•) and strong data augmentation a(•), such as synonym replacement and back translation, respectively. the weakly augmented data are then forwarded to two differently initialized models f and g to make predictions. high-confidence predictions that pass the adaptive local threshold τ t (c) are selected as pseudo-labels. especially, the pseudo-labels generated by one model are used to teach its peer network, i.e., cross-labeling. the unlabeled loss l u is computed between generated pseudo-labels and model predictions of strongly augmented data. here, we weigh more disagreement data (where both networks have different label predictions) to keep the two networks diverged but also allow the utilization of agreement data that are more likely weakly augmented data are first fed into two differently initialized models. predictions with confidence surpassing adaptive local thresholds are selected as pseudo labels for training. notably, pseudo labels from one model are used to teach its peer network, i.e, cross-labeling. unlabeled data loss is then computed between pseudo labels from weakly-augmented data and model predictions from strongly augmented data. here, we weigh more disagreement data to keep the two models diverged for effective mutual learning. to be correct. a complete algorithm for jointmatch is presented in algorithm 1. next, we explain our key components in detail. ## conclusion in this paper, we propose jointmatch, a semisupervised text classification approach that integrates ideas from recent semi-supervised learning and learning with noise. our method is motivated by observed problems in semi-supervised text classification (sstc): model bias towards easy classes and error accumulation. jointmatch utilizes adaptive thresholding, cross-labeling, and weighted disagreement & agreement updates to address these issues effectively. through extensive experiments on three standard sstc benchmark datasets, we found that jointmatch significantly outperforms previous works on all datasets and demonstrates surprising improvement over fixmatch on an extremely scarce label setting. we also show that jointmatch can generalize well to datasets that have a large number of classes.