SAT : Improving Semi-Supervised Text Classification with Simple Instance-Adaptive Self-Training (2022)

Abstract: Self-training methods have been explored in recent years and have exhibited great performance in improving semi-supervised learning. This work presents a simple instance-adaptive self-training method (SAT) for semi-supervised text classification. SAT first generates two augmented views for each unlabeled data, and then trains a meta learner to automatically identify the relative strength of augmentations based on the similarity between the original view and the augmented views. The weakly-augmented view is fed to the model to produce a pseudo-label and the strongly-augmented view is used to train the model to predict the same pseudo-label. We conducted extensive experiments and analyses on three text classification datasets and found that with varying sizes of labeled training data, SAT consistently shows competitive performance compared to existing semi-supervised learning methods.

## introduction pretrained language models have achieved extremely good performance in a wide range of natural language understanding tasks @xcite . however, such performance often has a strong dependence on large-scale high-quality supervision. since labeled linguistic data needs large amounts of time, money, and expertise to obtain, improving models' performance in few-shot scenarios (i.e., there are only a few training examples per class) has become a challenging research topic. semi-supervised learning in nlp has received increasing attention in improving performance in few-shot scenarios, where both labeled data and unlabeled data are utilized @xcite @xcite . recently, several self-training methods have been explored to obtain task-specific information in unlabeled data. uda @xcite applied data augmentations to unlabeled data and proposed an unsupervised consistency loss that minimizes the divergence between different unlabeled augmented views. to give self-training more supervision, mixtext @xcite employed mixup @xcite to learn an intermediate representation of labeled and unlabeled data. both uda and mixtext utilized consistency regularization and confirmed that such regularization exhibits outstanding performance in semi-supervised learning. to simplify the consistency regularization process, fixmatch @xcite classified two unlabeled augmented views into a weak view and a strong view, and then minimized the divergence between the probability distribution of the strong view and the pseudo label of the weak view. however, in nlp, it is hard to distinguish the relative strength of augmented text by observation, and randomly assigning an augmentation strength will limit the performance of fixmatch on text. to tackle this problem in fixmatch, our paper introduces an instance-adaptive self-training method sat, where we propose two criteria based on a classifier and a scorer to automatically identify the relative strength of augmentations on text. our main contributions are: ## related work our work combines data augmentation and consistency regularization to improve semi-supervised learning and is inspired by prior work in these areas. recently, several papers have proposed reinforcement learning policies @xcite and curriculum learning strategies @xcite to automatically augment data. also, a couple of consistency regularization methods are introduced to simplify the semi-supervised learning process @xcite as well as to boost performance in domain adaptation scenarios @xcite to improve semi-supervised learning. as far as we know, our work is the first to apply a meta-learner to automatically determine the augmentation strength in consistency regularization in semi-supervised text classification. ## conclusion in closing, this paper has proposed an instanceadaptive self-training method sat to boost performance in semi-supervised text classification. inspired by fixmatch, sat combines data augmentation and consistency regularization and designs a novel meta-learner to automatically determine the relative strength of augmentations. empirical experiments and ablation studies confirm sat is simple yet effective in improving semi-supervised learning.