Rethinking Semi-supervised Learning with Language Models (2023)

Abstract: Semi-supervised learning (SSL) is a popular setting aiming to effectively utilize unlabelled data to improve model performance in downstream natural language processing (NLP) tasks. Currently, there are two popular approaches to make use of the unlabelled data: Self-training (ST) and Task-adaptive pre-training (TAPT). ST uses a teacher model to assign pseudo-labels to the unlabelled data, while TAPT continues pre-training on the unlabelled data before fine-tuning. To the best of our knowledge, the effectiveness of TAPT in SSL tasks has not been systematically studied, and no previous work has directly compared TAPT and ST in terms of their ability to utilize the pool of unlabelled data. In this paper, we provide an extensive empirical study comparing five state-of-the-art ST approaches and TAPT across various NLP tasks and data sizes, including in- and out-of domain settings. Surprisingly, we find that TAPT is a strong and more robust SSL learner, even when using just a few hundred unlabelled samples or in the presence of domain shifts, compared to more sophisticated ST approaches, and tends to bring greater improvements in SSL than in fully-supervised settings. Our further analysis demonstrates the risks of using ST approaches when the size of labelled or unlabelled data is small or when domain shifts exist, and highlights TAPT as a potential solution.

## introduction pre-training (pt) language models (lms) @xcite @xcite over large amounts of text data (e.g. with masked language modelling) and then fine-tuning on task-specific labelled data offer large performance gains across nlp tasks. semi-supervised learning (ssl) @xcite @xcite is a powerful and effective approach to utilize unlabelled data. a typical ssl setting assumes access to a (relatively small) labelled training set and an (often large) unlabelled set. the goal of ssl is to make effective use of the unlabelled data to improve model (i.e. lms) performance. in nlp, self-training (st) approaches have been proposed to produce pseudo labels for unlabelled examples to train the model (e.g. in @xcite . with the advent of neural networks, st approaches typically focus on using student-teacher models to assign pseudolabels to the unlabelled data (e.g. in @xcite @xcite @xcite . apart from the sophisticated st approaches, @xcite proposed task adaptive pre-training (tapt), which is a straightforward yet effective method for utilising unlabelled examples. this method involves continuing pre-training the lm on the task-specific data without using labels, before proceeding with fully-supervised fine-tuning. tapt and st are both motivated by the need for effectively leveraging unlabelled examples, raising the questions of how tapt performs in ssl tasks, as well as how these two approaches perform against each other. in this work, we investigate the performance of tapt against five state-of-the-art st approaches across five nlp tasks ( §4). we empirically show that tapt outperforms all state-of-the-art st approaches on several tasks, suggesting that it should serve as a strong baseline for ssl methods. previous research @xcite has shown that tapt can improve performance in fullysupervised settings. our study goes further by showing that tapt can be even more effective in ssl settings ( §4). we next study the impact of using different amounts of labelled and unlabelled data for ssl ( §5). our experiments show that st approaches are prone to suffering from insufficient labelled or unlabelled data, while tapt is more robust across different combinations of labelled and unlabelled data sizes. contrary to the common assumption that tapt requires a large amount of data to perform well (e.g. @xcite , our results show that tapt improves performance with just a hundred unlabelled samples. we conduct further analysis on the impact of domain shifts in labelled or unlabelled data. while st approaches generally suffer from domain shifts, tapt is more robust and even benefits from domain shifts ( §6). in summary, the main contributions of this paper are as follows: ## st vs tapt ## exploring the limits of st and tapt in §4, our experimental results showed inconsistent results across datasets. for example, st performs better on imdb while tapt achieves better results on amazon review and yahoo! answer. we hypothesize that this might be attributed to the exposure to different sizes of labelled or unlabelled data. to verify this hypothesis and shed light on the differences in performance between datasets, we compare tapt and st (using adamatch and flexmatch as representatives) by sampling different labelled and unlabelled sizes in imdb, sst-2, amazon review and yahoo! answer. figure 3 visualizes the differences in performance between tapt and st, where each cell represents the macro-f 1 performance difference of tapt over st (averaged across five seeds). in each case, the highest performance among flexmatch and adamatch is selected to represent the performance of st. overall, we observe that: (1) tapt improves the fine-tuning performance even with a few hundred unlabelled examples; and (2) tapt performs more stable across the different labelled and unlabelled data sizes than st approaches. below we provide a comprehensive analysis of the impact of labelled and unlabelled sizes. #1. tapt works even with a few hundred unlabelled samples. it is generally assumed that tapt requires a large amount of unlabelled data to perform well (e.g. @xcite . however, we surprisingly observe that tapt can bring substantial improvement over supervised baseline even with a relatively small number of unlabelled samples, as shown in figure select the performance of tapt and supervised from the first column (the lowest unlabelled size) for each dataset in figure 3 and plot their average performance over different labelled sizes. figure 4 shows that tapt improves over the supervised baseline with just one hundred or one thousand samples. for instance, tapt achieves a 5.5% increase in f 1 score compared to the supervised baseline when using only 1k unlabelled samples on yahoo! answer. additionally, this performance is achieved without the need for large amounts of tokens in each sample, as training samples from sst-2, on average, contain only 9 tokens and training samples from yahoo! answer contain about 32 tokens (see examples in to further quantify the model performance in case of scarce unlabelled and adequate labelled data, we choose the three lowest unlabelled sizes (the first three columns) excluding the lowest labelled size (the last row) in figure 3 for each dataset. our analysis shows that st has 67%, 56% and 54% probability of falling below the super-vised baselines on sst-2, amazon review, and yahoo! answer respectively. even on imdb where st generally performs well, it still has a probability of 33% to fall behind super-vised. in contrast, tapt never performs worse than supervised in those cases. we provide computation details and comparative statistics in appendix §c. the specific value of adequate unlabelled size boundary for st approaches depends on the nature of the dataset as well as the labelled size. figure 5 illustrates that as the size of the labelled data increases, st approaches require more unlabelled data to surpass the supervised baselines. for example, on amazon review, st trained with 100 labelled samples requires about 5k unlabelled samples to perform better than supervised, while st trained with 10k labelled samples requires about 100k unlabelled samples. adjusting the unlabelled size accordingly might be conducive to exploiting the full potential of st approaches. both st and tapt have demonstrated the ability to exploit unlabelled data in this setting. figure 3 shows that st dominates in imdb when more than 10 labelled and 100 unlabelled samples are available. on the other hand, tapt generally performs better than st on amazon review and yahoo! answer, indicating that the answer to which approach is better depends on the nature of the dataset and task. as labelled and unlabelled data size increase, the difference between st and tapt shrinks (colours fade and lines converge in figures 3 and 5 ). as the labelled data in size reaches the unlabelled data, the method of st reduces to fully-supervised, which is generally outperformed by tapt @xcite . ## domain adaptation we next investigate how st and tapt compare in the presence of domain shifts between labelled and unlabelled data in two additional settings (refer to ## related work leveraging unlabelled data by continuing pretraining. previous work has shown that further pre-training lms on the unlabelled data of a task (e.g. @xcite @xcite or in-domain data (e.g. @xcite @xcite is beneficial to downstream tasks. however, it is unknown whether this is valid in ssl settings. previous studies in computer vision @xcite and speech recognition @xcite ) have compared pt and st. however, our study has a different focus, specifically, we compare tapt and st in nlp tasks. concurrently to our work, @xcite put forward prompt-based continued pre-training, which primarily aims to enhance the performance of promptbased fine-tuning techniques @xcite . this approach outperforms these state-of-the-art st approaches @xcite @xcite as well as the conventional cls-based fine-tuning with tapt. semi-supervised learning. recent work in ssl has demonstrated great progress in effectively exploiting unlabelled data. a wide range of approaches has been proposed including pseudo labeling @xcite , temporal ensemble @xcite , mean teacher @xcite , virtual adversarial training @xcite , fixmatch @xcite . a major issue for st approaches is confirmation bias, where the student model would accumulate errors from the teacher model when learning with inaccurate pseudo-labels (e.g. @xcite @xcite . while many efforts towards st (e.g. @xcite @xcite @xcite @xcite have been made in nlp, the performance of st approaches across various labelled and unlabelled sizes has yet to be thoroughly explored. although mukherjee and awadallah (2020); @xcite noted that training st approaches from tapt checkpoints can improve the performance, the performance of tapt in ssl tasks has not been either well-researched by previous works or compared with state-of-the-art st approaches. ## conclusion in this work, we shed light on how tapt performs against state-of-the-art st approaches in various ssl settings. our experiments reveal that tapt achieves strong and robust performance, even with just a few hundred unlabelled examples. we further demonstrate that the st approaches are vulnerable to small amounts of either labelled or unlabelled data. we also find that tapt is more robust than st approaches in joint domain adaptation and ssl settings. overall, our empirical study demonstrates that tapt is a strong ssl learner, competitive to more sophisticated st approaches. in future work, we plan to further explore the potential of tapt with unsupervised learning signals.