Weaker Than You Think: A Critical Look at Weakly Supervised Learning (2023)

Abstract: Weakly supervised learning is a popular approach for training machine learning models in low-resource settings. Instead of requesting high-quality yet costly human annotations, it allows training models with noisy annotations obtained from various weak sources. Recently, many sophisticated approaches have been proposed for robust training under label noise, reporting impressive results. In this paper, we revisit the setup of these approaches and find that the benefits brought by these approaches are significantly overestimated. Specifically, we find that the success of existing weakly supervised learning approaches heavily relies on the availability of clean validation samples which, as we show, can be leveraged much more efficiently by simply training on them. After using these clean labels in training, the advantages of using these sophisticated approaches are mostly wiped out. This remains true even when reducing the size of the available clean data to just five samples per class, making these approaches impractical. To understand the true value of weakly supervised learning, we thoroughly analyze diverse NLP datasets and tasks to ascertain when and why weakly supervised approaches work. Based on our findings, we provide recommendations for future research.

## introduction weakly supervised learning (wsl) is one of the most popular approaches for alleviating the annotation bottleneck in machine learning. instead of collecting expensive clean annotations, it leverages weak labels from various weak labeling sources such as heuristic rules, knowledge bases or lowerquality crowdsourcing @xcite . these weak labels are inexpensive to obtain, but are often noisy and inherit biases from their sources. deep learning models trained on such noisy data without regularization can easily overfit to the noisy labels @xcite . many advanced wsl techniques have recently been proposed to combat the noise in weak labels, and significant progress has been reported. on certain datasets, they even manage to match the performance of fully-supervised models @xcite @xcite . in this paper, we take a close look at the claimed advances of these wsl approaches and find that the benefits of using them are significantly overestimated. although they appear to require only weak labels during training, a substantial number of clean validation samples are used for various purposes such as early-stopping @xcite and meta-learning @xcite @xcite . we cast doubt on this practice: in real-world applications, these clean validation samples could have instead been used for training. to address our concern, we explore fine-tuning models directly on the validation splits of eight datasets provided by the wrench benchmark @xcite and compare it to recent wsl algorithms. the results are shown in figure 1 . interestingly, although all wsl models generalize better than the weak labels, simply fine-tuning on the validation splits outperforms all wsl methods in almost all cases, sometimes even by a large margin. this suggests that existing wsl approaches are not evaluated in a realistic setting and the claimed advances of these approaches may be overoptimistic. in order to determine the true benefits of wsl approaches in a realistic setting, we conduct extensive experiments to investigate the role of clean validation data in wsl. our findings can be summarized as follows: altogether, we show that existing wsl approaches significantly overestimate their benefits in a realistic setting. we suggest future work to (1) fully leverage the available clean samples instead of only using them for validation and (2) consider the simple baselines discussed in this work when comparing wsl approaches to better understand wsl's true benefits. ## overall setup problem formulation. formally, let x and y be the feature and label space, respectively. in standard supervised learning, we have access to a training set @xmath0, y i )} n @xmath1, y ) ∈ x × y. in weak supervision, we are instead given a weakly labeled dataset d @xmath2, ŷi )} n @xmath3, where ŷi represents labels obtained from weak labeling sources such as heuristic rules or crowd-sourcing. foot_0 ŷi is noisy, i.e., it may be different from the ground-truth label y i . the goal of wsl algorithms is to obtain a model that generalizes well on d test ∼ d c despite being trained on d w ∼ d n . in recent wsl work, a set of clean samples, d v ∼ d c , is also often included for model selection. 3 datasets. we experiment with eight datasets covering different nlp tasks in english. concretely, we include four text classification datasets: (1) agnews @xcite , ( 2 ) @xcite , (3) yelp @xcite , (4) trec @xcite , two relation classification datasets: (5) semeval @xcite and ( 6 ) chemprot @xcite , and two named-entity recognition (ner) datasets: (7) conll-03 @xcite ) and ( 8 ) @xcite . the weak annotations are obtained from the wrench @xcite 4 is clean data necessary for wsl? recent best-performing wsl approaches rely on a clean validation set for model selection. figure 1 reveals that they fail to outperform a simple model that is directly fine-tuned on the validation set. therefore, a natural question to ask is: "will wsl still work without accessing the clean validation set?". if the answer is yes, then we can truly reduce the burden of data annotation and the benefits of these wsl approaches would be undisputed. this section aims to answer this question. setup. we compare three different validation choices for model selection using either (1) a clean validation set from d v as in prior work, (2) weak labels from dv obtained by annotating the validation set via weak labeling sources (the same procedure used to construct training annotations), or (3) no validation data at all. in the last setting, we randomly select 5 sets of hyperparameters from our search space (see appendix c). we run the wsl approaches introduced in section 3 on all eight datasets with different validation choices and measure their test performance. each experiment is repeated 5 times with different seeds. while one may expect a certain drop in performance when switching from d v to dv , the absolute performance of a model does not determine the usefulness of a wsl method. we are more interested in whether a trained model generalizes better than the weak labels. 5 in realistic applications, it is only worth deploying trained models if they demonstrate clear advantages over the weak labels. therefore, we report the relative performance gain of wsl approaches over the weak labels. formally, let p w l , p α denote the performance (accuracy, f1-score, etc.) achieved by the weak labels and a certain wsl method α, respectively. the the relative performance gain is defined as g α = (p α -p w l )/p w l . we consider a wsl approach to be effective and practically useful only if g α > 0. results. figure 2 shows the relative performance gain for all considered wsl approaches. when model selection is performed on a clean validation set (green curve), all weak supervision baselines generalize better than the weak labels. sophisticated methods like cosine and l2r push the performance even further. this observation is consistent with previous findings @xcite . however, when using a weakly labeled validation set (yellow curve), all wsl baselines become ineffective and barely outperform the weak labels. more interestingly, models selected through the weakly labeled validation sets do not outperform models configured with random hyperparameters (purple curve). these results demonstrate that model selection on clean validation samples plays a vital role in the effectiveness of wsl methods. without clean validation samples, existing wsl approaches do not work. now that we know clean samples are necessary for wsl approaches to work, a follow-up question would be: "how many clean samples do we need?" intuitively, we expect an improvement in performance as we increase the amount of clean data, but it is unclear how quickly this improvement starts to level off, i.e., we may find that a few dozen clean samples are enough for wsl approaches to perform model selection. the following section seeks to answer this question. figure 3: the impact of the number of clean validation samples on performance. we plot average performance and standard deviation over 5 runs varying the size of the clean validation data. whenever a small proportion of validation data is provided, most wsl techniques generalize better than the weak label baseline (grey dashed line). performance improves with additional validation samples, but this tendency usually levels out with a moderate number of validation samples. setup. we apply individual wsl approaches and vary the size of clean data sub-sampled from the original validation split. for text and relation classification tasks, we draw an increasing number of clean samples @xmath4, 10, 15, 20, 30, 40, 50} per class when applicable. 6 in the case of ner, as a sentence may contain multiple labels from different classes, selecting exactly n samples per class at random is impractical. hence, for ner we sample @xmath5, 100, 200, 300, 400, 500} sentences for validation. for each n , we run the same experiment 5 times. note that the clean data is used solely for model selection in this set of experiments. results. as shown in figure 3 , in most cases, a handful of validation samples already make wsl work better than the weak labels. we observe an increasing trend in performance with more validation samples, but typically this trend weakens with a moderate size of samples (~30 samples per class or ~200 sentences) and adding more samples provides little benefit. there are a few exceptions. for example, on imdb all methods except l2r consistently perform better with more validation data. on conll-03, on the other hand, most methods seem to be less sensitive to the number of samples. overall, the results suggest that a small amount 6 the validation set of semeval is too small to support @xmath6. also, if a dataset is unbalanced, we randomly select n × c samples, where c denotes the number of classes. this is a realistic sampling procedure when performing data annotation. the previous sections have shown that current wsl approaches (1) do not improve over direct finetuning on the existing validation splits (figure 1 ) and ( 2 ) require only a small amount of validation samples to be effective (figure 3 ). this section investigates whether the conclusion from figure 1 would change with less clean data, i.e., can wsl approaches outperform direct fine-tuning when less clean data is available? setup. we follow the same procedure as in section 5 to subsample the cleanly annotated validation sets and fine-tune models directly on the sampled data. in addition to the standard fine-tuning approach @xcite , we also experiment with three parameter-efficient fine-tuning (peft) approaches as -in the few-shot setting -they have been shown to achieve comparable or even better performance than fine-tuning all parameters @xcite @xcite . in particular, we include adapters @xcite , lora @xcite , and bitfit @xcite . figure 4: using clean data for validation vs. training. we show the average performance (acc. and f1-score in %) difference between (parameter-efficient) fine-tuning approaches and cosine when varying amounts of clean samples. cosine uses the clean samples for validation, whereas fine-tuning approaches directly train on them (indicated in the legend with the subscript 'c'). for most sequence classification tasks, fine-tuning approaches work better once 10 clean samples are available for training. for ner, several hundreds of clean sentences may be required to attain better results via fine-tuning. refer to appendix d for a comparison with other wsl approaches. we use one fixed set of hyperparameter configurations and train models for 6000 steps on each dataset. 7 we report performance at the last step and compare it with wsl approaches which use the same amount of clean data for validation. results. figure 4 shows the performance difference between the fine-tuning baselines and co-sine, one of the best-performing wsl approaches, when varying the number of clean samples. it can be seen that in extremely low-resource cases (less than 5 clean samples per class), cosine outperforms fine-tuning. however, fine-tuning approaches quickly take over when more clean samples are available. lora performs better than co-sine on three out of four text classification tasks with just 10 samples per class. agnews is the only exception, where cosine outperforms lora by about 1% when 20 samples per class are available, but adapters outperform cosine in this case. relation extraction has the same trend where 10-20 samples per class are often enough for fine-tuning approaches to catch up. for ner tasks, all finetuning approaches outperform cosine with as 7 the hyperparameters are randomly picked from the ranges mentioned in the original papers of corresponding methods and fixed across all experiments. we did not cherrypick them based on the test performances. in most cases the training loss converges within 300 steps. we intentionally extend training to show that we do not rely on extra data for early-stopping. we find that overfitting to the clean data does not hurt generalization. a similar observation is made in @xcite . detailed configurations are presented in appendix d. few as 50 sentences on conll-03. ontonotes seems to be more challenging for fine-tuning and 400 sentences are required to overtake cosine. still, 400 sentences only account for 0.3% of the weakly labeled samples used for training cosine. this indicates that models can benefit much more from training on a small set of clean data rather than on vast amounts of weakly labeled data. note that the fine-tuning approaches we experiment with work out-of-the-box across nlp tasks. if one specific task is targeted, few-shot learning methods with manually designed prompts might perform even better. 8 hence, the performance shown here should be understood as a lower bound of what one can achieve by fine-tuning. nevertheless, we can see that even considering the lower bound of fine-tuning-based methods, the advantage of using wsl approaches vanishes when we have as few as 10 clean samples per class. for many realworld applications, this annotation workload may be acceptable, limiting the applicability of wsl approaches. the wsl approaches have only used clean samples for validation so far, which is shown to be inefficient compared to training directly on them. we question whether enabling wsl methods to further fine-tune on these clean samples would improve their performance. in this section, we study a straightforward training approach that makes use of both clean and weak labels. 9 setup. given both the weakly labeled training data and a small amount of clean data, we consider a simple two-phase training baseline. in the first phase, we apply wsl approaches on the weakly labeled training set, using the clean data for validation. in the second phase, we take the model trained on the weakly labeled data as a starting point and continue to train it on the clean data. we call this approach continuous fine-tuning (cft). in our experiment, we apply cft to the two bestperforming wsl approaches, cosine and l2r, along with the most basic wsl baseline, ft w . we sample clean data in the same way as in section 5. the training steps of the second phase are fixed at 6000. each experiment is repeated 5 times with different seeds. ## conclusions and recommendations our extensive experiments provide strong evidence that recent wsl approaches heavily overestimate their performance and practicality. we demonstrated that they hinge on clean samples for model selection to reach the claimed performance, yet models that are simply trained on these clean samples are already better. when both clean and weak labels are available, a simple baseline (ft w +cft) performs on par with or better than more sophisticated methods while requiring much less computation and effort for model selection. inspired by prior work @xcite , our recommendations for future wsl approaches are the following: we hope our findings and recommendations will spur more robust future work in wsl such that new methods are truly beneficial in realistic lowresource scenarios.