ULF : Unsupervised Labeling Function Correction using Cross-Validation for Weak Supervision (2023)

Abstract: A cost-effective alternative to manual data labeling is weak supervision (WS), where data samples are automatically annotated using a predefined set of labeling functions (LFs), rule-based mechanisms that generate artificial labels for the associated classes. In this work, we investigate noise reduction techniques for WS based on the principle of k-fold cross-validation. We introduce a new algorithm ULF for Unsupervised Labeling Function correction, which denoises WS data by leveraging models trained on all but some LFs to identify and correct biases specific to the held-out LFs. Specifically, ULF refines the allocation of LFs to classes by re-estimating this assignment on highly reliable cross-validated samples. Evaluation on multiple datasets confirms ULF’s effectiveness in enhancing WS learning without the need for manual labeling.

## introduction a large part of today's machine learning success rests upon large amounts of annotated training data. however, collecting manual annotation (even in a reduced amount, e.g., for fine-tuning large pretrained models @xcite , active learning @xcite , or semi-supervised learning @xcite ) is tedious and expensive. an alternative approach is weak supervision (ws), where data is labeled in an automated process using one or multiple ws sources such as keywords @xcite , knowledge bases @xcite , and heuristics @xcite , which are encoded as labeling functions @xcite . lfs are applied to unlabeled datasets to obtain weak training labels, which are cheap, but often conflicting and error-prone, requiring improvement (see we focus on enhancing the quality of weak labels using k-fold cross-validation. intuitively, by in contrast to correcting labels as in previous work for supervised settings, we utilize the crossvalidation principle in a ws setting to adjust and improve the lf-to-class assignment. in some cases, an lf correctly captures some samples but mislabels others. for example, one of the lfs used to annotate the youtube dataset @xcite is the keyword my, which is effective in identifying spam messages such as subscribe to my channel or check my channel out (see to address such cases, we introduce a new method ulf: unsupervised labeling function correction (figure 1 ), which comprises both erroneous labels detection and correction by leveraging weakly supervised knowledge. ulf re-estimates the joint distribution between lfs and class labels during cross-validation based on highly confident class predictions and their co-occurrence with matching lfs. importantly, this reestimation is performed out-of-sample, meaning it is guided by the data itself without involving additional manual supervision. instead of a hard assignment of naive ws (i.e., an lf either corresponds to the class or not), ulf performs a fine-adjusted one, which helps to correct the label mistakes. for ex-ample, such assignment reduces the association of the lf "my" with the spam class, resulting in dominant ham probability in samples 2 and 3 in figure 1 . moreover, ulf successfully labels samples with no lfs matched, as, e.g., sample 4, unlike other methods that filter them out @xcite . we conduct extensive experiments using feature-based and pre-trained models to demonstrate the effectiveness of our method. to the best of our knowledge, we are the first to adapt crossvalidation denoising methods to ws problems and refine the lfs-to-class allocation in the ws setting. ## related work weak supervision (ws) has been widely applied to different tasks across various domains, such as text classification @xcite , relation extraction @xcite , named entity recognition @xcite , video analysis @xcite , medical domain @xcite , image classification @xcite . weak annotations are easy to obtain but prone to errors. approaches to improving noisy data include building a specific model architecture @xcite , using additional expert annotations @xcite , identifying and removing or downweighting harmful samples @xcite , or learning from manual user guidance @xcite . ulf is compatible with any classifier and do not require any manual supervision; instead of removing the samples, ulf corrects the labels, utilizing as much ws data as possible. k-fold cross-validation, a reliable method for assessing trained model quality @xcite , is also often used to detect errors in manual annotations @xcite @xcite , but has not been applied to a ws setting. we propose ws extensions to some of these methods in appendix b and use cross-validation in ulf. ## ulf: unsupervised labeling function correction in this section, we present the key elements of ulf. more details can be found in appendix a, the pseudocode is provided in algorithm 1. given a dataset @xmath0, x 2 , ..., x n } to be used for k-class classifier training. in ws setting, we do not have any gold training labels, but only a set of lfs @xmath1, l 2 , ..., l l }. an lf l j matches a sample x i if some condition formulated in l j holds for x i . following sedova et al. (2021), we store this information in a binary matrix z n ×l , where z ij @xmath2. a set of lfs matched in sample x i is denoted by l x i , where l x i ⊂ l and |l x i | ∈ [0, |l|] the lfs to class correspondence is stored in a binary matrix t l×k , where t @xmath3.e., l i assigns the samples to the class j) foot_2 . the weak training labels ỹ = {ỹ 1 , ỹ2 , ..., ỹn }, ỹj ∈ k are obtained by multiplying z and t , apply majority vote, and break the ties randomly. the main goal of the ulf algorithm is to refine the t matrix. the graphical explanation is provided in figure 2 . first, class probabilities are predicted for each sample using k-fold cross-validation on the training set x and weak labels ỹ . we propose different ways of splitting the data into k folds f 1 , ..., f k , with the most reliable method being splitting by signatures (refer to appendix a for details and other possible splitting methods). the samples' signatures, i.e., the sets of lfs matched in each sample, are collected, split into k folds, and used to create data folds: x train @xmath4, x out @xmath5. next, k models are separately trained on each of k-1 folds and applied to the held-out folds, resulting in out-of-sample predicted probabilities p n ×k . the out-of-sample label ŷi for each sample i is determined by selecting the class with the highest probability, provided that this highest probability exceeds the class average threshold t j : that is, a sample x i is confidently assigned to class j if the out-of-sample probability of it belonging to class j is higher than the average out-of-sample probability of all samples initially assigned to this class. if no probability exceeds the class thresholds (e.g., all probabilities are equally small), the sample is disregarded as unreliable for further calculations. these assignments are used to build an lfs-toclasses confidence matrix c l×k , which estimates the joint distribution between matched lfs and predicted labels. for each lf l i and each class k j , the confidence matrix c l×k is populated by counting the number of samples that have lf l i matched and confidently assigned to class k j : algorithm 1: ulf: unsupervised labeling function correction for weak supervision input: unsupervised training data x, samples to lfs matrix z n ×l , lfs to classes matrix qlm,ŷ j . this calibration ensures that ql×k sums up to the total number of training samples, and the sum of counts for each lf is the same as in the original z matrix; thus, ql×k can be utilized as a crossvalidated re-estimation of t . finally, a refined t * matrix is calculated as follows: here, the hyperparameter p, @xmath6, 1], determines the extent to which information from the original t matrix should be retained. the resulting t * matrix is utilized to generate improved labels for additional ulf iterations or training the final classifier. unlabeled samples. ulf also takes advantage of unlabeled samples that do not have any lfs matched. a portion of these samples, determined by the hyperparameter λ, is randomly labeled and included in cross-validation training, with reestimation in subsequent iterations. to leverage all unlabeled samples in a fine-tuning-based setting, we also include the optional cosine self-training step @xcite , which can be executed during cross-validation and/or final classifier training. cv / final youtube spouse trec sms yorùbá hausa ft_ft 96.8 22.0 68.2 96.1 54.6 43.0 ft_cos 94.4 36.9 76.6 96.2 54.2 48.2 cos_ft 95.2 21.3 68.6 96.1 55.8 43.6 cos_cos 94.8 33.0 76.8 96.1 54.2 44.5 ## conclusion & future work in our work, we focused on denoising ws data by leveraging information from lfs. our approach assumes that the noise specific to some lfs can be detected by training a model that does not use those lfs signals and then comparing its predictions to the labels generated by the held-out lfs. this idea is used in our method ulf, which improves the weak labels based on the data itself, without leveraging external knowledge. extensive experiments validate the effectiveness of our approach and support our initial hypothesis of the significant role of lfs in denoising ws data. in future work, we plan to try ulf for other tasks, such as sequence tagging and image classification, and perform more experiments on weakly supervised datasets with different peculiarities and in different languages.