Learning to Robustly Aggregate Labeling Functions for Semi-supervised Data Programming (2022)

Abstract: A critical bottleneck in supervised machine learning is the need for large amounts of labeled data which is expensive and time-consuming to obtain. Although a small amount of labeled data cannot be used to train a model, it can be used effectively for the generation of humaninterpretable labeling functions (LFs). These LFs, in turn, have been used to generate a large amount of additional noisy labeled data in a paradigm that is now commonly referred to as data programming. Previous methods of generating LFs do not attempt to use the given labeled data further to train a model, thus missing opportunities for improving performance. Additionally, since the LFs are generated automatically, they are likely to be noisy, and naively aggregating these LFs can lead to suboptimal results. In this work, we propose an LF-based bi-level optimization framework WISDOM to solve these two critical limitations. WISDOM learns a joint model on the (same) labeled dataset used for LF induction along with any unlabeled data in a semi-supervised manner, and more critically, reweighs each LF according to its goodness, influencing its contribution to the semi-supervised loss using a robust bi-level optimization algorithm. We show that WISDOM significantly outperforms prior approaches on several text classification datasets.

## introduction supervised machine learning approaches require large amounts of labeled data to train robust machine learning models. human-annotated gold labels have become increasingly important to modern machine learning systems for tasks such as spam detection, (movie) genre classification, sequence labeling, etc. the creation of labeled data is, however, a time-consuming and costly process that requires large amounts of human labor. together with the * equal contribution heavy reliance on labeled data for training models, this serves as a deterrent to achieving comparable performance on new tasks. as a result, various methods such as semi-supervision, distant supervision, and crowdsourcing have been proposed to reduce reliance on human annotation. in particular, several recent data programming approaches @xcite @xcite have proposed the use of human-crafted labeling functions to weakly associate labels with the training data. typically, users encode supervision as rules/guides/heuristics in the form of labeling functions (lfs) that assign noisy labels to the unlabeled data, thus reducing dependence on human-labeled data. the noisy labels were aggregated using label aggregators, which often employ generative models, to assign a label to the data instance. examples of label aggregators are snorkel @xcite and cage @xcite . these models provide consensus on the noisy and conflicting labels assigned by the discrete lfs to help determine the correct labels probabilistically. we could use the obtained labels to train any supervised model/classifier and evaluate on a test set. apart from the cascaded approach described above, recently proposed semi-supervised paradigm @xcite learns to aggregate labels using both features and a very small labeled set in addition to labeling functions. such approaches have been shown to outperform the completely unsupervised data programming approaches described above. data programming (unsupervised or semisupervised) requires carefully curated lfs, generally expressed in the form of regular expressions or conditional statements. even though creating lfs can potentially take less time than creating large amounts of supervised data, it requires domain experts to spend considerable time identifying and determining the patterns that should be incorporated into lfs. in this paper, we circumvent the requirement of human-curated lfs by instead automatically generating human-interpretable lfs as compositions of simple propositions on the data set by leveraging snuba @xcite which utilizes a small labeled-set to induce lfs automatically. however, as we will show, snuba suffers from two critical limitations, which keep it from outperforming even a simple supervised baseline that is trained on the same labeled-set. first, snuba only uses the labeled-set to generate the lfs but does not make effective use of it in the final model training. secondly, as it naively aggregates these lfs, it is not able to distinguish between very noisy lfs and more useful ones. this work addresses both of these limitations. in as a solution, how should be given less importance due to its noisy and conflicting nature, whereas how long, associated with the numeric label, should be given higher importance. in this paper, we present a bi-level optimization framework for reweighting the induced lfs, which effectively reduces the weights of noisy labels while simultaneously increasing the weights of the more useful ones. in figure 1 , we present an overview of our approach. we leverage semi-supervision in the feature space for more effective data programming using the induced (automatically generated) labeling functions. to enable this, we split the same labeled-set (which was used to generate the lfs) into a supervised set and validation set. the supervised set is used for semi-supervised data programming, and validation set is used to tune (reweight) the lfs. as a basic framework for semi-supervised data programming, we leverage spear @xcite , which has achieved state-of-theart performance. while the semi-supervised data programming approach helps in using the labeled data more effectively, it does not solve the problem of noise associated with the lfs. to address this, we propose an lf reweighting framework, wisdom foot_0 , which learns to reweight the labeling functions, thereby helping differentiate the noisy lfs from the cleaner and more effective ones. the reweighting is achieved by framing the problem in terms of bi-level optimization. we argue that using a small labeled-set can help improve label prediction over hitherto unseen test instances when the labeled-set is bootstrapped for (i) inducing lfs, (ii) semi-supervision, and (iii) bi-level optimization to reweight the lfs. for most of this work, the lfs are induced automatically by leveraging part of the approach described in @xcite . the lfs are induced on the entire labeledset, whereas the semi-supervision and reweighting are performed on the supervised set and validation set respectively (which are disjoint partitions of labeled-set). our contributions are as follows: while leveraging snuba @xcite only for au-youtube trec 60 80 86.0 58.3 57.2 50.5 83.1 59.0 82.1 59.2 90.4 60.1 94.1 63.3 supervised snuba auto-spear vat wisdom hum-spear figure 2: a summary plot contrasting the performance gains obtained using wisdom on previous state-of-theart approaches on youtube and trec (using lemma features). wisdom outperforms other learning approaches with auto-generated lfs. ## experiments we present evaluations across six datasets that we describe in the following section 4.1. in ## related work in this section, we describe some additional related work that was not covered in section 2. ## conclusion while induction of labeling functions (lfs) for data-programming has been attempted in the past by @xcite , we observe in our experiments that the resulting model in itself does not perform well on text classification tasks, and turns out to be even worse than the supervised baseline. a more recent semi-supervised data programming approach called spear @xcite , when used in conjunction with the induced lfs, performs better, though it fails to consistently outperform the supervised baseline. in this paper, we introduce wisdom, a bi-level optimization formulation for reweighting the lfs, which injects robustness into the semi-supervised data programming approach, thus allowing it to perform well in the presence of noisy lfs. on a reasonably wide variety of text classification datasets, we show that wisdom consistently outperforms all other approaches, while also coming close to the skyline of spear using human-generated lfs. the keyword how or many. another example is among lfs associated with location class, lfs city and country were given higher weightage than where. however, wisdom does a poor job by assigning a very small weight value to the single lf who associated with the human class. in in