S ep LL : Separating Latent Class Labels from Weak Supervision Noise (2022)

Abstract: In the weakly supervised learning paradigm, labeling functions automatically assign heuristic, often noisy, labels to data samples. In this work, we provide a method for learning from weak labels by separating two types of complementary information associated with the labeling functions: information related to the target label and information specific to one labeling function only. Both types of information are reflected to different degrees by all labeled instances. In contrast to previous works that aimed at correcting or removing wrongly labeled instances, we learn a branched deep model that uses all data as-is, but splits the labeling function information in the latent space. Specifically, we propose the end-to-end model SepLL which extends a transformer classifier by introducing a latent space for labeling function specific and task-specific information. The learning signal is only given by the labeling functions matches, no pre-processing or label model is required for our method. Notably, the task prediction is made from the latent layer without any direct task signal. Experiments on Wrench text classification tasks show that our model is competitive with the state-of-the-art, and yields a new best average performance.

## introduction in recent years, large language modelling approaches have proven their applicability to a wide range of tasks, mainly due to the pre-training and fine-tuning paradigm. this has created a need for large labeled datasets, as training on these datasets enables models to achieve state-of-the-art performance. however, obtaining manually created labels is expensive, tedious and often requires expert knowledge. as a consequence, significant areas of research are devoted to addressing this challenge by minimizing the need for labeled data. for example, research directions include transfer learning @xcite or few-shot learning @xcite . another research direction to address this challenge is weakly supervised learning. the idea is to use human intuitions, heuristics and existing resources, e.g., related databases, to create weak (noisy) labels. several approaches have been proposed to increase the quality of the resulting labels. for example, @xcite use generative modeling to learn a probability distribution over the labeling function matches, i.e., weak labels, and unknown true labels in order to denoise the labels and subsequently train a classifier. recently, several works use student-teacher schemes that use knowledge inherent to pre-trained models @xcite @xcite . usually a summary statistic of weak labels, such as majority vote, is used as ground truth and iteratively updated during training, for example by employing a regularization based on the prediction confidence of the model @xcite . thus, most methods share the property that the weak labels, i.e., the learning signals, are transformed or updated throughout the learning process. instead of updating the weak labels, we want to keep them as-is and make use of a different intuition. each labeling function provides information relevant to the prediction task but also information only related to the function itself. our idea is to view these two types of information as complementary and build a model which separates them. to this end, we propose sepll, an end-to-end model that stacks two branched latent layers, representing target-task-related and labeling-functionrelated information, on top of a transformer encoder and recombines them for predicting labeling function occurrences (figure 1 ). then, the learning signal is only given by the weak labels. notably, the task prediction is performed from the latent space without any direct supervision. multiple information routing strategies are employed to improve the separation. in order to evaluate the performance, experi-ments on the text classification tasks of the wrench benchmark @xcite are performed. our model achieves state-of-the-art performance when compared to standalone models as well as when combined and compared with the self-improvement method cosine @xcite . an ablation study shows the importance of each information routing strategy. the experiments show that in addition to its task performance, the model is able to memorize the labeling function information. the contributions can be summarized in three parts: 1) we introduce a new intuition about the information provided by labeling functions and turn it into a method, sepll, reflecting the intuition in the latent space. 2) we provide an analysis through experiments on the wrench benchmark, an ablation study and an in depth analysis of the two latent spaces. 3) we provide the code and a suitably transformed version of the input data. 1 2 related work weak supervision. a main concern in machine learning is that a large amount of labeled data is needed in order to train models that achieve stateof-the-art performance. among others, the field of weak supervision aims to address this issue. the idea is to formalize human knowledge or intuitions into weak supervision sources, called labeling functions, which can be used to produce weak labels. examples of labeling functions are heuristic rules, e.g., keywords, regular expressions, other pre-trained classifiers or knowledge bases in distant supervision @xcite @xcite . a main challenge that appears in a weak supervision setting is how to create accurate labeling functions and how to unify and denoise them. majority vote, snorkel @xcite (based on data programming) and flying squid @xcite are methods that compute weak labels based on generative models over the labeling function matches and unknown true labels. these models are referred to as label models. subsequently so called end-models, e.g., bert-style classifiers @xcite , or methods dedicated to noisy training labels are used to train a final model. recently, neural methods, including the use of pre-trained models, gained more traction. cachay et al. (2021) use a classifier and a probabilistic 1 https://github.com/andst/sepll encoder for the labeling function matches and optimize them using a noise-aware loss. similarly, @xcite combine a classifier and a attentionbased denoiser, but also include unlabeled samples. @xcite introduced cosine, which is a method to self-optimize classification models. they leverage contrastive learning and confidence regularization, i.e., high-confidence samples, to optimize a model's performance. other approaches use additional signals. for instance, implyloss @xcite ) uses access to exemplars, i.e., single, correctly labeled samples and astra @xcite follows an attention based student-teacher mechanism with an additional supervision of a few manually annotated labeled samples. @xcite uses a meta self-refinement approach which makes use of access to the validation performance. our experiments are built on the weak supervision benchmark (wrench) @xcite , which is a framework that aims to provide a unified and standardized way to run and evaluate weak supervision approaches. a wide range of tasks, datasets and implementations of weak supervision methods are available. ## method the motivation of this work is that each labeling function provides two types of information. on the one hand, it provides information about the target task, e.g., spam detection, and on the other hand it provides information related to the labeling function itself. this translates to our model, called sepll, which aims to separate these two types of signals in a latent space. figure 1 provides an overview of sepll. in this section, we first introduce some notation and then describe the architecture of sepll. following, the training mechanisms, which aim to support the separation of the two information types are discussed. ## experimental setup before analysing the experiments, we discuss the experimental setup. more specifically, an overview of the used datasets, baselines, hyperparameters and some notes on reproducibility and implementational details are given. ## experiments this section provides an analysis of the capabilities of the model. after the general performance analysis, an ablation study of the information strategies is performed. furthermore, we investigate how much information flows through the path associated with the latent class prediction, and how much though the other, lf-specific path. ## conclusion this work tackles weak supervision in a novel way. instead of denoising the weak labels or iteratively updating them, the weak labels are used as the training target as-is. we introduce the sepll model, which separates information relevant for the target task through the usage of two latent spaces. one latent space is used to perform downstream task prediction, the other one aims to keep the labeling function specific information. thus, the prediction is made from the latent space without any direct supervision. experiments show that the model is able to achieve state-of-the-art performance. an additional investigation shows that the model is able to memorize the labeling function information. hence, the model cannot only be used for downstream tasks but also to predict labeling function matches.