Understanding Programmatic Weak Supervision via Source-aware Influence Function
Abstract
Programmatic Weak Supervision (PWS) aggregates the source votes of multiple weak supervision sources into probabilistic training labels, which are in turn used to train an end model.
With its increasing popularity, it is critical to have some tool for users to understand the influence of each component ( e.g. , the source vote or training data) in the pipeline and interpret the end model behavior.
To achieve this, we build on Influence Function (IF) and propose source-aware IF, which leverages the generation process of the probabilistic labels to decompose the end model’s training objective and then calculate the influence associated with each (data, source, class) tuple.
These primitive influence score can then be used to estimate the influence of individual component of PWS, such as source vote, supervision source, and training data.
On datasets of diverse domains, we demonstrate multiple use cases: (1) interpreting incorrect predictions from multiple angles that reveals insights for debugging the PWS pipeline, (2) identifying mislabeling of sources with a gain of 9%-37% over baselines, and (3) improving the end model’s generalization performance by removing harmful components in the training objective (13%-24% better than ordinary IF).
Introduction
One of the major bottlenecks for deploying modern machine learning models is the need for a substantial amounts of human-labeled training data, often annotated over long periods of time and at great expense.
As machine learning models become increasingly powerful but also data hungry, new “data-centric” AI development workflows and systems have emerged, wherein the labeling and development of this training data is positioned and supported as the central development activity.
One recent and increasingly popular type of data-centric AI development uses Programmatic
Weak Supervision (PWS) , wherein users focus on developing a diversity of noisy, programmatic supervision sources [32, 33, 48, 45] to programmatically annotate training data in an efficient way.
Specifically, these weak supervision sources , e.g. , heuristics, knowledge bases, and pre-trained models, are often abstracted as labeling functions (LFs) [33], which is a user-defined program that provides potentially noisy labels for some subset of the data.
So far, different modeling techniques have been developed to aggregate the noisy votes of LFs to produce training labels (often referred to as a label model )
Finally, these training labels are in turn used to train an end model for the downstream classification tasks.
In this study, we focus on this two-stage PWS pipeline [45].
With the increasing popularity of PWS in various applications [13, 36, 40, 19], it is of great importance to provide practitioners with a tool that helps them understand the behavior of a trained end model as an effect of each upstream component ( e.g. , source vote, training data, LFs, etc. ) involved in the model’s training process.
First, by understanding what are the deciding factors that lead to a model’s specific prediction, users are able to verify whether such influence is desirable in terms of critical aspects such as model safety and fairness [11, 35, 16].
Secondly, since developing LFs is still inevitably a partly manual process in real-world applications and may involve human expert in an iterative process [5, 20, 49], it is beneficial to offer users with feedback of how individual component (like each LF) influences the end model performance on the downstream task.
Finally, the rendered understanding of (in)efficacy of the PWS pipeline can be naturally exploited to automatically improve the end model performance.
Towards the goal of developing tools to help understand the model behavior, Influence Function (IF) [22, 15, 8] has been recently applied to interpret the model prediction as influence of the training data for large-scale applications.
However, most of existing IF studies are unaware of the generation process of training labels like PWS and only estimate the influence of the training data [22, 3].
Notably, the group IF study [23] has attempted to apply IF on PWS, but it simply estimates the influence of each LF as summation of data influence and is limited to a specified PWS without learnable label model.
In this work, we aim to develop a general framework that helps estimate the influence of each PWS’s component on the end model and is aware of (yet computationally agnostic to) the choice of label model.
To achieve this, we leverage the knowledge of how the probabilistic labels are synthesized from sources and accordingly decompose the training loss of each data into multiple terms, each of which is associated with what we called ( i, j, c ) -effect , i.e. , the effect of j -th LF on c -th dimension (corresponding to class c ) of the i -th data’s probabilistic label.
With such a training loss decomposition, we propose two means of calculating our source-aware IF , namely, reweighting and weight-moving, based on different perturbations on the training respectively.
Each estimated source-aware influence score can be interpreted as the influence of removing the ( i, j, c ) -effect from the pipeline.
By simply aggregating over specific dimension(s), it can be utilized to estimate the influence of different components of PWS including (1) each source vote, (2) each LF and (3) each training data, since the influence score is additive [23].
In experiments, we exploit the source-aware IF in a diversity of 13 classification datasets including tabular, text, and image data, and present multiple applications.
First, we show that source-aware IF can help to understand the behavior of the end model in terms of the PWS components; in particular, when interpreting the same incorrect prediction made by two PWS pipelines with different label models, data-level IF might lead to the same influential training data, whereas source-aware IF is capable of revealing the most responsible LF or source vote.
Second, we use source-aware IF as a tool to identify mislabeling of each LF and show that it outperforms baselines with a significant margin of 9%-37%.
Finally, we demonstrate that the training loss decomposition and source-aware IF enable fine-grained training loss perturbation and consequently lead to better test loss improvement (13%-24% improvement over ordinary IF [22] and group IF [23]).
To summarize, this work makes the following contributions.
First , we propose source-aware IF tailored for PWS that incorporates our knowledge of the generation process of training labels.
Second , we develop two means of calculating the source-aware IF based on different training loss perturbations.
Third , we demonstrate a variety of use cases of the source-aware IF and conduct extensive quantitative and qualitative experiments on real datasets in different domains to verify the efficacy of source-aware IF on understanding and improving PWS pipelines.
Preliminary
We denote scalars and generic items as lowercase letters, vectors as lowercase bold letters, and matrices as bold uppercase letters.
We use superscripts to index generic items, e.g. , the i -th data x ( i ) , and use subscripts to index certain dimensions of a vector, e.g. , the i -th component of a vector v i , or the vector output of a function f ( · ) i .
All derivation details are deferred to the appendix.
Programmatic Weak Supervision.
We target at a C -way classification problem and have a training dataset of size N , i.e. , D
=
x ( i ) , y ( i ) }
i ∈
[ N ] ,
where x ( i ) ∈
X
= R d
and y ( i )
∈ Y =
[ C ] .
The ground truth y ( i ) s are not observed.
We have M labeling functions (LFs) { λ ( j ) ( · ) } j ∈ [ M ] .
Each λ ( j ) ( · ) ∈ {− 1 } ∪ Y ( j ) , where Y ( j ) ⊂
Y .
That is, each λ ( j ) ( · ) either abstains ( − 1 ) or outputs a specific label y ∈ Y ( j ) on a given data point.
We define the label matrix as L ij = λ ( j ) ( x ( i ) ) , and with a slight abuse of notation, use L ( x ( i ) ) to represent the row of L corresponding to all of the labels assigned to x ( i ) .
The goal of Programmatic Weak Supervision (PWS) is to train an end classifier h θ : X → Y with { x ( i ) }
i ∈
[ N ] and L only, where h θ ( x ) = arg max c ∈
[ C ] f θ ( x ) c and f θ ( x ) ∈ R C is a vector-valued function parameterized by θ .
A typical PWS pipeline involves a label model g W ( L ( x )) parametrized by W , which aims to produce a weak probabilistic label ˆ y ( x ( i ) )
=
g W ( L ( x ( i ) )) ∈ ∆ C for each data point x
( i ) (denoted as ˆ y ( i ) for simplicity).
The label model produces a dataset ˆ D
= { x ( i ) , ˆ y ( i ) }
i ∈
[ N ] , which is then used to train f θ via the noise-aware loss [33] ˆ ℓ (ˆ y ( i ) , f θ ( x ( i ) ))
=
E y ∼ ˆ y
( i )
[ ℓ ( y, f θ ( x ( i ) )]
= (cid:80) Cc
=
ˆ y
( i ) c ℓ ( c, f θ ( x ( i ) )) , where ℓ ( · , · ) is the cross-entropy loss, leading to the following training objective:
[Equation]
where θ ⋆ is the minimizer of the loss ˆ L ( ˆ D ; θ ) over the training set ˆ D .
Influence Function.
The Influence Function (IF) estimate how the minimizer θ ⋆ of the empirical loss ℓ would change if we were to reweight the i -th training example
z ( i ) =
( x ( i ) , y ( i ) ) by ϵ i .
The key idea is to make a first-order Taylor approximation of the changes in θ ⋆ around ϵ i = 0 .
Specifically, if the i -th training sample is upweighted by a small ϵ i , the perturbed risk minimizer θ ⋆ϵ i becomes:
[Equation]
The changes of the model parameters due to the introduction of the weight ϵ i are:
[Equation]
where H θ ⋆ = 1 N (cid:80) i ′ ∇ 2 θ ℓ (cid:16) y ( i ′ ) , f θ ⋆ ( x ( i ′ ) ) (cid:17) is the Hessian of the objective at θ ⋆ and assumed to be positive definite [22].
Similarly, the change of loss with respect to a single sample z ′ =
( x ′ , y ′ ) is
[Equation]
Then, as shown in previous work [24], the change of loss over a holdout set , e.g. , the validation set
D v , is simply the sum of the changes over every sample in the set
[Equation]
The change of loss w.r.t.
one sample is often interpreted as its effect on model predictions over the training set [22], while the change of loss over a holdout set can be leveraged to improve the test loss by discarding/downweighting the training examples with negative impact [38, 42, 24].
Considering the extraordinary influence that might be caused by mislabeled examples, the relative influence function (RelatIF)
[3] has been proposed to measure the influence of a sample relative to its global effects.
The RelatIF can be defined as ϕ ′
i
[Equation]
=
[Equation]
Methodology
Given an unlabeled training set { x ( i ) } i ∈ [ N ] and M labeling functions (LFs) { λ ( j ) ( · ) } j ∈ [ M ] , we study the scenario where a Programmatic Weak Supervision (PWS) pipeline is employed to synthesize probabilistic labels { ˆ y ( i ) } i ∈ [ N ] for training an end model f θ ( x ) for a C -way classification problem.
We aim for a general framework for explaining and debugging the PWS pipeline through Influence
Function (IF)
[22], which estimates the influence of each training data on test loss/prediction for a better understanding of model behavior and is able to improve/debug the training data accordingly.
In the PWS pipeline, we have full knowledge of how training labels are synthesized from different sources, i.e. , LFs, which opens the opportunity to trace the influence in a source-aware way.
In this section, we propose the source-aware IF tailored for PWS and describe its applications.
3.1 Training loss decomposition
By examining the popular choices of label models g W ( · ) [9, 33, 31], we show that their parameters can be unifiedly represented as a tensor W ∈ R M × ( C +1)
× C , where the middle ( C + 1) dimension size is due to the additional "abstention" label LFs output.
The generation process of the probabilistic labels can then be expressed as
[Equation]
where σ ( · ) could be (1) the identity function σ id ( x ) =
x for the classic Majority Voting label model or its variants, or (2) the exponential function σ exp ( x ) = exp ( x ) to recover the softmax function used in statistical models such as Dawid-Skene model [9] and Snorkel MeTaL [31].
Intuitively, label models aim to learn a source-specific weight for each LF to aggregate the votes in an unsupervised manner.
Details can be found in Appendix A.
In both cases, for the i -th data, a single parameter
W j, L ij ,c controls the effect of the j -th LF’s vote L ij on the c -th dimension of the probabilistic label, i.e. , ˆ y ( i ) c .
For convenience, we denote the effect of L ij on ˆ y ( i ) c as the ( i, j, c ) -effect , as it uniquely corresponds to the i -th data, the j -th LF and c -th dimension of the aggregated probabilistic label.
For a better understanding, we visualize how a probabilistic training label is generated and the ( i, j, c ) -effect in Figure 1.
With such unified formulation of probabilistic label generation, we rewrite the noise-aware loss [33] as
[Equation]
We proceed with the σ ( · ) being the identity function because it enables the loss decomposition introduced below.
We will later discuss other choices of σ ( · ) .
By simple rearrangement, we could decompose the loss over the training set into the summation of all N × M × C terms:
[Equation]
Such a decomposition is source-aware in the sense that it preserves how the source votes are aggregated via the label model, and each ¯ ℓ i,j,c ( θ ) indeed corresponds to the ( i, j, c ) -effect.
3.2 Source-aware Influence Function
As a consequence of training loss decomposition, we are now able to compute the influence score for each term ¯ ℓ i,j,c ( θ ) , which is the influence of removing ¯ ℓ i,j,c ( θ ) or, in other words, the ( i, j, c ) -effect from training.
We provide two means of calculating the influence score: reweighting and weightmoving , which correspond to two different ways of perturbing the training loss.
Specifically, given an (unseen) instance z ′ =
( x ′ , y ′ ) and the cross-entropy loss ℓ ( y ′ , f θ ( x ′ )) , we are interested in the influence score defined by the change of loss ℓ ( y ′ , f θ ( x ′ ))
caused by the differences in the model parameters θ after perturbing the training loss ˆ L ( ˆ D ; θ ) .
We first give the formulation of an ordinary
IF for a training data z ( i ) =
( x ( i ) , ˆ y ( i ) ) in the context of PWS, which is smilar to Eq. 4:
[Equation]
Here, H − 1 θ ⋆ = 1 N
[Equation]
Ni =1 ∇ 2 θ ˆ ℓ (ˆ y
( i ) , f θ ⋆ ( x ( i ) ))
and θ ⋆ is the minimizer of the training loss before the perturbation.
We reuse the notation of H θ ⋆ and θ ⋆ in this section as their meanings are unchanged.
Reweighting.
To study the influence of removing the ( i, j, c ) -effect, we reweight ¯
[Equation]
i,j,c ( θ ) such that
[Equation]
Then, the influence score, i.e. , the difference of loss on z ′ =
( x ′ , y ′ ) caused by reweighting the loss term ¯ ℓ i,j,c ( θ ) , is,
[Equation]
In contrast to ordinary IF that reweights the loss w.r.t.
individual training data, the proposed one is finer-grained since it reweights the decomposed loss terms.
By reweighting, the resultant probabilistic training label is no longer sum-to-one and the solution (Eq. 11) is only for identity σ ( · ) function.
Weight-moving.
We also explore an alternative way of perturbing the training loss called weight- moving.
Different from reweighting, its computation is agnostic to the σ ( · ) function and the probabilistic training label is still sum-to-one.
Specifically, we remove the ( i, j, c ) -effect in the probabilistic training label ˆ y ( i ) and renormalize it, leading to a new probabilistic training label:
[Equation]
Then, we perturb the training loss by moving the weight of the original loss on the i -th data ˆ ℓ (ˆ y ( i ) , f θ ( x ( i ) )) to a new loss term ˆ ℓ (ˆ y
( i
[Equation]
− j
[Equation] [Equation]
[Equation]
When ϵ i,j,c = − 1 N , the loss term ˆ ℓ (ˆ y
( i ) , f
[Equation] [Equation]
i ) )) is indeed replaced by the new loss ˆ ℓ (ˆ y
( i ) − j ′ c ′ , f θ ( x ( i ) )) .
We again provide the formulation of corresponding influence score:
[Equation] [Equation]
Discussion.
Note that the change of loss over a holdout set D v can be similarly computed and we denote it as ¯ ϕ rwi,j,c
( D v ) and ¯ ϕ wmi,j,c
( D v ) .
We omit the input variable and the superscript and use ¯ ϕ i,j,c to represent the proposed source-aware IF in general.
The ¯ ϕ i,j,c corresponds to the influence of a specific perturbation on the training loss: setting the label model parameter W j, L ij ,c to zero for the i -the data so that the ( i, j, c ) -effect is removed from training.
Then, the difference between reweighting and weight-moving reduces to whether renomarlization on the probabilistic labels is performed after the perturbation.
Extension to RelatIF [3].
We additionally demonstrate how to compute the RelatIF [3] in our source-aware
IF setting, Specifically, the self-influence is ¯ ϕ selfi,j,c = −∇ θ ¯ ℓ i,j,c ( θ ⋆ )
⊤ H − 1 θ ⋆ ∇ θ ¯ ℓ i,j,c ( θ ⋆ ) .
Then, the source-aware RelatIF for both ¯ ϕ rwi,j,c and ¯ ϕ wmi,j,c are ¯ ϕ rwi,j,c (cid:113) ¯ ϕ selfi ,j,c and ¯ ϕ wmi,j,c (cid:113) ¯ ϕ selfi ,j,c , respectively.
3.3 Use cases
Case 1: estimating influence of different components.
One major benefit of our training loss decomposition and the identity σ ( · ) function is that we could readily reuse the source-aware IF ¯ ϕ
i,j,c
[Table]
to estimate different components of the PWS pipeline, e.g. , individual data or LF.
This is because according to the study of group influence [23], when measuring the change in test prediction/loss, the influence is additive, i.e. , the influence of a set is the sum of influences of its constituent points.
However, such a nice property only holds true for the reweighting method, while, in theory, the influence calculated by weight-moving does not follow due to the fact that they are not additive.
We list the influence of different components of a PWS pipeline as well as how to calculate them using the fine-grain IF ¯ ϕ i,j,c in Table 1.
Note that the 1st and 2nd row of Table 1 indeed recover the ordinary IF of a training data [22] and a LF [23], respectively.
Case 2: improving test loss.
Another application of IF is to improve the test loss by identifying and then discarding/downweighting the harmful training data [24, 42, 38].
Similarly, the source-aware IF ¯ ϕ i,j,c can be employed to improve the test loss by discarding/downweighting the loss term ¯ ℓ i,j,c ( θ ) with negative impact.
Specifically, we have the following theorem for reweighting (a similar result for weight-moving can be found in Appendix E.2):
Theorem 1.
Discarding or downweighting the loss terms in S − = { ¯ ℓ i,j,c ( · ) | i ∈
[ N ] , j ∈ [ M ] , c ∈
[ C ] , ¯ ϕ rwi,j,c ( D t ) > 0 } from training could lead to a model with lower loss over a holdout set D t :
[Equation]
where θ ⋆ S − is the optimal model parameters obtained after the perturbation.
In practice, we use the validation set D v and S − ( α ) =
{ ¯ ℓ i,j,c ( · ) | i ∈
[ N ] , j ∈ [ M ] , c ∈
[ C ] , ¯ ϕ rwi,j,c
( D v ) > α } to tolerate the estimation error of IF where the hyperparameter α controls the proportion of items to be perturbed [24].
Note that this method of improving test loss relies on the training loss decomposition and σ ( · ) being the identity function.
3.4 Beyond the identity function
The aforementioned use cases rely on the additivity of influence score and the training loss decompo- sition, which are both consequences of σ ( · ) being the identity function.
When σ ( · ) is the exponential function, we could approximate the label model g W ( · ) by a new label model ¯ g ¯ W ( · ) with parameter ¯
W of the same shape but using identity function instead.
Obviously, we want the new label model ¯ g ¯ W ( · ) to reproduce the same probabilistic labels as those generated by g W ( · ) .
This motivates us to solve the following optimization problem for ¯ W :
[Equation]
Then, we can replace the label model g W ( · ) with ¯ g ¯ W ⋆ ( · ) everywhere.
As a consequence, we are now able to unify various types of label models in a single framework and enjoy the convenience and advantages brought by the linearity of the identity function.
Note that for the original label model g W ( · ) with σ ( · ) being the exponential function, we can still derive the influence
¯ ϕ
i,j,c (see appendix).
We study the effect of replacing original label model with the approximated one on the end model performance in the experiment section.
Experiment 4.1 Setup
Datasets.
We include the following four classification datasets in WRENCH [48], a collection of benchmarking datasets for PWS: Census , Youtube , Yelp , and IMDb .
Note that the Census dataset is a tabular dataset while the others are textual datasets.
We use the labeling functions (LFs) released by
WRENCH [48].
We also include the following tabular datasets: Mushroom [12], Spambase [12], and
PhishingWebsites (PW) [29], for which we follow the instruction in the WRENCH [48] codebaseto generate LFs from a decision tree learned on the labeled data.
Finally, we follow [28] to derive
LFs for a multiclass image classification task using the DomainNet [30] dataset, which containsclasses of images in 6 different domains: real, sketch, quickdraw, painting, infograph and clipart.
We construct a classification task for each domain with the 5 classes containing the largest number of instances.
We then train classifiers using the selected classes within the remaining five domains as LFs.
We name the 6 resultant datasets DN-real , DN-sketch , DN-quickdraw , DN-painting ,
DN-infograph , and DN-clipart for each domain, respectively.
Following [48], we use a pre-trained
BERT model [10] to extract features for textual data and use ResNet-18 [17] pre-trained on ImageNet for image data.
Model Training and Evaluation.
For label model, we focus on three commonly-used choices:
Majority Voting (MV), Dawid-Skene model (DS) [9], and Snorkel [31].
The σ ( · ) function of the latter two label models is the exponential function and we use their approximated variants (as mentioned in
Section 3.4) for all experiments.
Following [23], we use logistic regression for the end model and defer the experiments on using neural networks to the appendix.
We train the end model initialized as zero by gradient descent for 10,000 epochs with learning rate being 0.001, and then calculate the influence score on validation set D v throughout the experiments.
When using the test loss as evaluation metric, we use D v to do model selection and report the test loss of the select model.
For simplicity, we use IF (R-IF) to represent ordinary IF (RelatIF) baseline; RW (R-RW) and
WM (R-WM) are the proposed source-aware IF (RelatIF) calculated by the reweighting and weightmoving method respectively.
We also use the term SIF
( R-SIF ) to represent the source-aware IF (RelatIF) when we do not care whether it is calculated by reweighting or weight-moving.
Additional experiments and details are in Appendix G.
4.2 Experimental results
Understand model behavior from multiple angles.
First and foremost, source-aware IF can be used to understand the end model behavior from multiple angles, while ordinary IF can only identify influential training data.
Here, we take the DN-real dataset as an example and use reweighting-based source-aware IF ( RW ) to diagnose misclassified test data.
For a single misclassified test data of the end model, we present the most "responsible" (1) training data, (2) LF, and (3) labeling (the vote of a
LF on a training data), where being the most "responsible" means having the highest influence score on that data and likely being the primary cause of the misclassification.
In Table 2, we show that for the same image misclassified by the end model trained with different label models, we could have totally different explanations and components of the PWS pipeline to blame on.
Specifically, for a “ dog ” image misclassified as “ golf club ”, we can see that for both the DS and Snorkel label models, the most "responsible" training image are the same but the most "responsible" LF and labeling are different: for DS the LF trained with images of painting domain is likely the primary cause for the misclassification, while for Snorkel we might blame more on the LF trained with clipart images.
Without source-aware IF, we can hardly distinguish the cause of misclassification for DS and Snorkel based solely on the training data as the most "responsible" one is identical.
Identify mislabeling of LFs.
One unique advantage of source-aware IF over ordinary IF is that it allows us to trace the influence of each LF’s labeling on a specific training data ( ϕ L ij = (cid:80) C c =
¯
ϕ
i,j,c , the 3rd row in Table 1), which is helpful in understanding the expertise of each individual LF and thus provides clues for practitioners to debug LFs, while ordinary IF can only estimate the influence of training data as a whole.
We use ϕ L ij as a scoring function of how likely the j -th LF mislabels
[Table] [Table]
[Figure]
i -th data, i.e. , L ij ̸
=
y ( i ) , to identify the mislabeling of individual LF.
We thus formulate it as a binary classification task and report the average precision (AP), a standard evaluation metric of a scoring function.
We use the discrepancy between j -th LF’s labeling on i -th data ( L ij ) and the probabilistic label output by (1) label model ( LM ), (2) end model ( EM ), and (3) a K -nearest neighbor classifier ( K = 10 ) trained on validation set ( KNN ) as baseline scoring functions.
Specifically, we calculated the discrepancy as one minus the dot product of L ij ’s one-hot representation and the probabilistic label; for example, if L ij ’s one-hot representation is [0 , 1 , 0]
⊤ and the probabilistic label is [0 . 5 , 0 . 4 , 0 .
3] ⊤ , the discrepancy is 1 − 1 × 0 .
4 = 0 .
Note that KNN is a more fair and competitive baseline because it directly leverages the validation set and, as our source-aware IF, does not introduce any parametrized model in addition to what PWS has.
We present the results in Table 3; we can see the superiority of source-aware IF in this task since it outperformed baselines in most cases and the averaged gains are about 9%-37% for different label models.
Intriguingly, when LM and EM render unsatisfactory performance, e.g. , on DomainNet , which indicates the quality of the label or end model is not good, the source-aware IF still performed well.
This means it is less sensitive to the quality of PWS, which is critical for debugging the pipeline.
Improve test loss.
We compared the proposed source-aware IF as well as its RelatIF variant to ordinary IF/RelatIF in terms of the capability of improving the test loss.
For source-aware IF/RelatIF, we discarded loss terms in S − ( α ) and then re-trained the end model (as described in Section 3.3), while for IF/RelatIF, we likewise discarded training data in D − ( α ) =
{ z ( i ) | i ∈
[ N ] , ϕ
i ( D v ) > α } .
For these methods, we tuned the hyperparameter α on validation set for best validation loss.
We additionally included the group influence ( GIF ) [23] that estimates the influence of each LF and is only applicable to MV as in [23]; we discarded k LFs with highest negative impact and tune k on validation set.
The results are in Table 4, where ERM (Emperical Risk Minimization) is the ordinary training without any perturbation.
From the results, we conclude that although ordinary
IF/RelatIF can already improve the test loss over ordinary training ( ERM ) by a large margin (>0.
test loss on average), the source-aware IF/RelatIF can further boost the performance by a similar
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
margin (>0.1 averaged test loss improvement over IF/RelatIF).
This shows the benefit of employing
IF in a source-aware manner and provides insights for practitioners to improve the PWS pipeline.
Discussion.
One may develop sophisticated methods to identify mislabeling of LFs or to improve test loss and achieve better performance than source-aware IF, but source-aware IF is designed as a tool of interpreting the end model prediction and when used in these applications, it does not introduce additional parameterized model.
Also, we believe that source-aware IF can inspire advanced methods in the future and be leveraged by them to further boost the performance on the task of identifying mislabeling of LFs or improving test loss.
4.3 Additional studies
Estimate ordinary data-level IF via source-aware IF.
As described in Section 3.3, we can use the source-aware IF calculated by the reweighting method to estimate the data-level influence score ( ϕ x ( i ) =
(cid:80) Mj
=1 (cid:80)
Cc =1 ¯ ϕ
i,j,c , the 1st row in Table 1), which coincides with the influence computed by ordinary IF in theory.
Thus, we are curious about how well the source-aware IF can be used to estimate the ordinary data-level IF.
Because IF is known to be noisy in the value but good in their ranking [4, 23], we instead study the ranking correlation between ϕ x ( i ) and the ordinary IF.
In Tablewe report the Spearman’s ranking correlation coefficient ( ≤ 1 ), where larger coefficients indicate better estimation of the ordinary IF based on ϕ x
( i ) in terms of the ranking.
From the results, we can see that the ϕ x ( i ) actually preserves the ranking of ordinary IF quite well as Spearman’s coefficients are from 0.71 to 0.99, which further demonstrates the efficacy of source-aware IF.
[Figure]
Efficacy of label model approximation.
Since we bypassed the non-linearity of exponential function in DS and Snorkel with their approximated variants (Section 3.4), it is natural to ask how well the approximated label model can perform and whether or not the approximation is well-haved.
To answer these questions, in Figure 2 we show that although the averaged test loss of ordinary training ( ERM ) and best of IF/R-IF
( Best IF/R-IF ) are slightly better than their counterpart of approximated label model ( Approx.
ERM and Approx.
Best
IF/R-IF ), the best averaged test loss obtained by SIF/R-SIF with the approximated label model ( Approx.
Best SIF/R-SIF ) is much lower.
Such an observation indicates that with the goal of improving test loss, it is beneficial to choose σ ( · ) to be the identity function so that the training loss can be decomposed and perturbed based on source-aware IF.
Related work
Programmatic Weak Supervision.
In the Programmatic Weak Supervision (PWS) literature [33, 45], popular choices of labeling functions (LFs) include user-crafted heuristics [32, 39, 44], pretrained models [2, 28], external knowledge bases [18, 26], and crowd-sourced labels [9, 25].
Recently, researchers have developed various label models [33, 31, 14, 40, 47, 43, 37, 46].
In the PWS pipeline, the behavior of the end model is directly dictated by the soft labels, wherein the soft labels themselves are also intermediate products of the label model and the LFs’ votes.
Thus, to interpret the trained end model and potentially debug the PWS workflow, it is critical to understand the influence of each PWS component on the end model’s performance, which has yet received adequate research attention.
Influence Function.
Among various ways to understand a model’s behavior through the training data points, Influence Function (IF) is an effective technique with a wide range of applications [22, 23, 16, 7].
In PWS, although one can directly apply IF to understand a model’s behavior through the weakly labeled training data, more insights may be unveiled by further investigations of how each
PWS component, e.g., LFs and label models, influences the end model.
In similar spirit to us, [6] developed multi-stage IF to trace a fine-tuned model’s behavior back to the pretraining data.
Perhaps most similarly to our work, [23] conducted initial exploration on using group influence to study how removing an LF from the PWS pipeline would affect the end model’s performance.
However, the group influence method is only applicable to a specific simplified setting of PWS.
Limitation and social impacts
Limitations.
Although we study several popular label models and show how to use approximated label model in case that users need to use a label model that is not included in this work, user may still encounter difficulties in real applications when using complicated label model.
Also, this work focuses on the two-stage PWS pipeline, while very recently there are some one-stage methods, to which our framework may not be applicable.
In the main body of the paper, we use logistic regression as end model in our experiments, while in appendix we present experimental results of a simple two-layer neural network.
But we do not include experiments on complicated deep learning models like Convolutional Neural Network.
Social impacts.
This work aims at help people understanding the Programmatic Weak Supervision and has the potential to resolve the bias in the generation process of training label, which might have positive social impact.
We do not foresee any form of negative social impact induced by our work.
Conclusion
In this work, by leveraging the knowledge of how probabilistic training labels are aggregated from different sources in the PWS pipeline, we have proposed source-aware IF that enables users to estimate the influence of different components ( e.g. , source vote, training data, LFs, etc. .)
in the pipeline on the end model’s behavior, and unlocked various practical use cases.
Specifically, we have demonstrated how the proposed source-aware IF can be used to (1) unveil the most influential training data point, LF, or a specific vote to a misclassification made by a model, (2) identify mislabeling of
LFs on certain data points, and (3) improve the end model’s test performance.
Acknowledgement
Thanks the anonymous reviewers for their helpful comments and suggestions.
We also thank Haojie
Jia for his help of experiments.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] See appendix.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] See appendix.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] See appendix.
(b) Did you include complete proofs of all theoretical results?
[Yes] See appendix.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] See supplemental materials.
(b) Did you specify all the training details (e
g., data splits, hyperparameters, how they were chosen)?
[Yes] (c) Did you report error bars (e g., with respect to the random seed after running experiments multiple times)?
[No] For experiments with randomness, we report averaged results over 5 runs.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See appendix.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] (b) Did you mention the license of the assets?
[Yes] (c) Did you include any new assets either in the supplemental material or as a URL?
[Yes] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[Yes] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[Yes] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
A Label model reparametarization and illustrations A.1 Majority Voting
The Majority Voting (MV) is the most intuitive algorithm for aggregate LFs’ annotations.
The MV methods can be formalize as
[Equation]
By letting W j,k,c = 1 { k = c } , we re-formulate the above as
[Equation]
A.2 Dawid-Skene model
The parameters of the Dawid-Skene(DS)
[9] model are given by
[Equation]
[Equation]
= number of times the j -th LF votes for l when true label is c number of data voted by j -th LF whose true label is c , (18) and p c , which is prior of p ( y = c ) .
Let n ( j ) i,l be the number of times j -th LF assigns l for data x
( i ) and T i,c be the indicator variables: if the true label of data x ( i ) is c , then T i,c
= 1 , otherwise 0 .
When the true label for all data are available, the likelihood is given by
[Equation] [Equation]
By the Bayes’ theorem:
[Equation]
We can re-write Eq.
21 as
[Equation] [Equation]
By omitting the last log p c term and letting W j, L ij ,c = (cid:80) C +1 l =1 (cid:16) n ( j ) i,l log π ( j ) c,l (cid:17) , we have ∀
[Equation]
∈ [ C ] ,
[Equation]
The additional log p c term can be absorbed in the summation by introducing an additional LF whose label space consists of only one element and the corresponding parameter in W is p =
[log ( p ( y = 1)) , log ( p ( y = 2)) , · · · , log ( p ( y = C ))]
⊤ .
We omit this case for simplicity.
A.3 Snorkel MeTaL
The parameters µ of Snorkel MeTaL [31] are given by
[Equation]
where T j,c,m ( j ) is the indicator, T j,c,m ( j )
= 1 { y = c, λ ( j ) = m ( j ) } .
Given the prior of p ( y ) , by the
Bayes’ theorem we have:
[Equation]
We can further infer that
[Equation]
By letting W j
[Equation]
k
[Equation]
c = log µ j,c,k p c and, again, omitting the log p c term similar to the above derivation for the DS model, we re-formulate the above as
[Equation]
A.4 More complicated label model
For more complicated label models that we cannot formulate the inferred label as Eq.
6, we can still use the approximation described in Section 3.4.
Consider a label model g ( L ( x ) , x ) ∈ F in arbitrary functional class F , e.g. , neural network, and having additional dependency on data feature x 3 , we can still approximate such complicated function with identity function-based label model ¯ g ¯ W ( x ) ( L ( x )) similar to the aforementioned one except that ¯ W ( x ) :
X → R M × ( C +1)
× C is a similarly complicated function, e.g. , neural network, that maps each data x ∈ X to a unique label model parameter ¯ W ( x ) .
We leave the exploration of more complicated form of label models into future work.
B Influence Function derivation: the reweighting method
Fellow the derivation of [22], we can directly compute the influence of fine-grained sample ( x ( i ) , 1 s i W j, L ij ,c ) through up-weighting its corresponding fine-grained level loss with ϵ i,j,c .
In the following, we explicitly define the reweighted loss function with different σ ( · ) function.
B.1 Case 1: identity function
We define the loss with reweighted sample as,
[Equation]
and the corresponding risk minimizer as θ ⋆ϵ
i,j,c .
According to the Equation (4), the influence on the loss of one test sample ( x k ,
y k ) ∈
D v is −∇ θ ˆ ℓ ( y k , f θ ⋆ ( x k ))
⊤ H − 1 θ ⋆ ∇ θ ¯ ℓ i,j,c ( θ ⋆ ) · ϵ i,j,c , where
H −
[Equation]
⋆ = 1 N (cid:80) i,
[Equation] [Equation]
∇
[Equation]
¯
[Equation]
c ( θ ⋆ ) .
The influence score on the loss of z ′ =
( x ′ , y ′ ) by discarding the loss term ¯ ℓ i,j,c ( θ ) is,
[Equation]
And the influence on the loss on the set D v can be defined as,
[Equation]
B.2 Case 2: exponential function
With exponential function, the label for sample x ( i ) can be presented as,
[Equation]
where c ∈ [ C ] .
For the sample x ( i ) , if the value of j -th label function on c -th class is upweighted with ϵ i,j,c , then the corresponding label can be written as,
[Equation]
We assume that (cid:80) Ck ′
=1 exp ( (cid:80) Mj ′
=
1 W j ′ , L ij ′ ,k ′ )
≫ exp ( W j, L ij ,c ) · ( exp ( ϵ i,j,c W j, L ij ,c ) − 1) .
Note, this assumption holds in general, because ϵ i,j,c is close to zero as we discussed in the Section 2.
Then we have,
[Equation]
Then
[Equation]
the reweighted risk over the whole training set becomes
[Equation]
We denote the minimizer of the reweighted risk ˆ L ϵ i,j,c ( ˆ D ; θ ) as θ rw ⋆ ϵ i,j,c .
Then the change of parameters can be presented as,
[Equation]
And the change with respect to ϵ i,j,c is,
[Equation]
According to the Talyor expansion
[Equation]
we have exp( x ) =
+ x + O ( x ) .
Then, we can obtain
[Equation]
The influence score on the loss of z ′ =
( x ′ , y ′ ) by discarding the loss term ¯ ℓ i,j,c ( θ ) is,
[Equation]
C Influence Function derivation: the weight-moving method
Instead of employing the decomposing loss function, we introduce a more general influence estimation method - weight-moving Influence
[Equation]
which get ride of the loss decomposition and approximation and is agnostic to the selection of σ ( · ) function.
As we introduced previously, the label of sample x ( i ) for each class c can be defined as:
[Equation] [Equation]
Further, we define the label after removing the output value of j ′ -th label function on the c ′ -th class for sample i as:
[Equation]
Similarly
[Equation]
we can also define the label vector ˆ y ( i ) − j ′ c ′ for the sample x ( i ) .
Then, we define the minimizer of the weight-moving loss as:
[Equation]
The change of parameters with respect to ϵ i,j,c can be written as,
[Equation]
With ϵ i,j,c = − 1 N
[Equation]
the weight for the sample ( x ( i ) , ˆ y ( i ) ) is moved to the the sample ( x ( i ) ,
ˆ y
( i ) − jc ) .
Then, the influence of the loss on sample z ′ =
( x ′ , y ′ ) by removing the output value of j -th label function on the c -th class for sample i is,
[Equation]
We then define the weight-moving influence over the set D v as,
[Equation]
[Equation]
D Connection between weight-moving and reweighting method
To show the connection between weight-moving influence ( ¯ ϕ wmi,j,c ) and reweighting influence ( ¯ ϕ rwi,j,c ), we firstly decompose the weight-moving loss (37).
For the identity function, we have:
[Equation] [Equation]
For simplicity, we denote C i
=
(cid:80) Ck =1 (cid:80)
Mj =
1 W j, L ij ,k and C ′ i
= (cid:80) Ck =1 (cid:80)
Mj ′
=1 1 { k ̸ = c ′ , j ̸ = j ′ } · W j ′ , L ij ′ ,k .
Note C
i
C i → 1 , with j → ∞ or C → ∞ .
Further, we assume C ′
i
C
i ≈ 1 .
Then, we have,
[Equation]
Note, the above loss function is same as the loss we defined in the Equation (27).
Therefore, the weight-moving influence, ¯ ϕ wmi,j,c
( D v ) , is an approximation of the reweighting influence, ¯ ϕ rwi,j,c
( D v ) , especially in the setting with large number of labeling function M and large number of class C .
E Proofs for theoretical analysis
We first show the correctness of Theorem 1, in which we employ the reweighting influence as a proxy to downweight or discard data samples.
Then, we show that the similar conclusion can be obtained with weight-moving influence as the proxy.
We summarized the conclusion for weight-moving influence into the Theorem 2.
E.1 Proofs for Theorem
Theorem 1.
Discarding or downweighting the loss terms in S − = { ¯ ℓ i,j,c ( · ) | i ∈
[ N ] , j ∈ [ M ] , c ∈
[ C ] , ¯ ϕ rwi,j,c ( D t ) > 0 } from training could lead to a model with lower loss over a holdout set D t :
[Equation]
where θ ⋆ S − is the optimal model parameters obtained after the perturbation.
As we derived in Appendix B.1, the reweighting influence represent that the change of loss over the set D t through upweighting the fine-grained sample ( x ( i ) ,
1 s i W j, L ij ,c ) by ϵ i,j,c ,
[Equation]
Through downweighting or discarding (note, discarding a sample is equal to downweighting it with ϵ i,j,c = − 1 /N )
the fine-grained samples with ¯ ϕ rwi,j,c
( D v ) > 0 , the change of the loss is equal to ϵ i,j,c · ¯ ϕ rwi,j,c < 0 .
As commonly assumed in the previous works [42, 24], the influence caused by reweighting different samples is independent.
Then, with ϵ i,j,c = − 1 /N , we can get the conclusion that L ( D t ; θ
[Equation]
E.2 Proofs for Theorem
Theorem 2.
Discarding or downweighting the training loss term in S − = { ¯ ℓ i,j,c ( θ ) | i ∈
[ N ] , j ∈ [ M ] , c ∈
[ C ] , ¯ ϕ wmi,j,c
( D v ) > α } from training could lead to a model with lower loss over a test set D t :
[Equation]
where α ∈ R + is a positive number close to 0 , and θ ⋆ S − is the optimal model parameters obtained after the perturbation
[Equation]
We first decompose the weight-moving loss in the following form,
[Equation]
where, we denote ˜ L i,j,c
= ( 1 c ′ i − 1 c i ) (cid:80)
Mj ′
=1 (cid:80)
Cc ′
=1 1 [ c ̸ = c ′ ∨
j ̸ = j ′ ] (cid:0) − W j ′ , L ij ′ ,c ′ log ( f θ ( x
[Equation]
i ) )
c ′ ) (cid:1) and C i
=
(cid:80) Ck =1 (cid:80)
Mj =
1 W j, L ij ,k and C ′ i
= (cid:80) Ck =1 (cid:80)
Mj ′
=1 1 [ k ̸ = c ′ , j ̸ = j ′ ] · W j ′ , L ij ′ ,k .
Note, as we discussed in the Appendix D, 1 c ′ i − 1 c
i is a small positive value and close to 0.
Therefore, ˜ L i,j,c ≪ ¯ ℓ i,j,c .
[Equation]
We assume (cid:80) ( x k ,y k
[Equation]
∈D t ∇ θ ˆ ℓ ( y k , f
[Equation]
) ⊤
[Equation]
⋆
[Equation]
˜ L i,j,c · ϵ i,j,c ≤ α , where α > 0 .
Then, through selecting the samples with ¯ ϕ wmi,j,c
( D v ) ≥ α , the reweighting influence term ¯ ϕ rwi,j,c
( D v ) ≥ 0 .
Then we can get the conclusion that by reweighting the training loss term in S − = { ¯ ℓ i,j,c ( θ ) | i ∈
[ N ] , j ∈ [ M ] , c ∈
[ C ] , ¯ ϕ rwi,j,c
( D v ) > α } from training, we have L ( D t ; θ ⋆ϵ ) − L ( D t ; θ ⋆ ) ≤ 0 .
F Computation detail of inverse Hessian
The estimation of influence score requires the computation of the inverse hessian.
The size of hessian matrix is propotional to the number of model parameters, thus directly computing the inverse hessian
H − 1 θ ⋆ is very expensive.
For the H − 1 θ ⋆ involved in the reweighting influence, Equation(11), and the weight-moving influence Equation (14), we employed the LiSSA
(Linear time Stochastic SecondOrder Algorithm) method [1], which provide an unbiased estimation of the Hessian-vector product through implicitly computing it with a mini-batch of samples.
As demonstrated in the previous works [4, 41], the stochastic method is efficient and relatively accurate for sample-wise influence estimation.
The algorithm can be summarized as:
• Step 1.
Let v :=
(cid:80) ( x ′ ,y ′ ) ∈D v ∇ θ ℓ ( y ′ , f θ ⋆ ( x ′ ))
(cid:0) (cid:80) ( x ′ ,y ′ ) ∈D v ∇ θ ℓ ( y ′ , f θ ⋆ ( x ′ )) for weightmoving method (cid:1) , and initialize the inverse HVP estimation H − 1 0 ,θ ⋆ v = v .
• Step 2.
For i ∈ { 1 , 2 , . . .
, J } , recursively compute the inverse HVP estimation using a batch size B of randomly sampled a data point ( x i ′ , y i ′ ) , H − 1 i,θ ⋆ v
[Equation]
(cid:16) I − ∇ 2 θ ℓ ( y ( i ′ ) , f θ ⋆ ( x ( i ′ ) )
(cid:17) H − 1 i − 1 ,θ ⋆ v , where J is a sufficiently large integer so that the above quantity converges.
• Step 3.
Repeat Step 1-2 T times independently, and return the averaged inverse HVP estimations.
For the computation of self-influence, the influence estimation is required for each training sample.
For real-world dataset, estimating each sample separately using the LiSSA method is intolerable.
Instead of applying the stochastic method for each training sample, we leverage the relation between
Hessian matrix and Fisher matrix, and use the K-FAC method [27] directly compute the inverse
Hessian matrix.
We refer interested readers to Barshan et al.
[3] for details regarding the K-FAC approximation for the computation of inverse Hessian.
G Experimental details and additional results G.1 Dataset statistics and implementation details
We summarize the dataset statistics in Table 6.
All of the involved datasets are either publicly available or will be released upon the acceptance of this paper.
All experiments ran on a machine with an Intel(R) Xeon(R) CPU E5-2678 v3 with a 126G memory and a GeForce GTX 1080Ti-11GB GPU.
All the code was implemented in Python and largely based on the WRENCH [48] codebase.
G.2 Estimate actual effect of LF via source-aware IF
As described in Section 3.3, we can use the source-aware IF calculated by the reweighting method to estimate the influence score of each LF ( ϕ λ ( j )
= (cid:80) Ni =1 (cid:80)
Cc =1 ¯ ϕ
i,j,c , the 2nd row in Table 1).
Here,
[Table]
we study how well these influence scores reflect the actual effect of each LF.
In Table 7, we again report the Spearman’s ranking correlation coefficient ( ≤ 1 ).
Please note that the estimated influence of each LF is more likely to be prone to noise since it involves more terms in summation than that of a training data.
Also note that here we compare the estimated influence against the actual effect, which is calculated by removing the loss terms associated with each LF and retraining the end model.
From the results, we can see that although there do exists cases where the results do not pass the significance test, the averaged ranking correlations are good (from 0.675-0.766).
Such observations indicate that source-aware IF could be useful when estimating the influence of each IF.
[Table]
[Figure]
G.3 Additional experiments on neural network
Although when the end model is neural network, the theory of IF breaks down [22], we are curious about whether the proposed source-aware IF is still effective.
Thus, we conduct the experiments of identifying mislabeling of LFs (see Table 8) and improving test loss (see Table 9) using two-layer neural network with the ReLU activation function as end model in this section.
For simplicity, we do not include the methods based on RelatIF.
From the results, we can see that, similar to the experimental results in the main body of this paper, source-aware IF outperforms baselines with a large margin.
This indicates that source-aware IF is still effective even when the theory does not hold.
G.4 The study of label model approximation
In this study, we take a close look at the label model approximation.
Specifically, we present the
Mean Squared Error (MSE) and expected disagreement (DE) between the output of label model
DS / Snorkel and their approximated version in Table 10.
The MSE is evaluated against the predicted label posterior of models, and the ED is the expectation of Y 1 ̸
= Y 2 over the training samples, where
Y 1 and Y 2 is the predicted label of the label model and its approximated version.
From the results, we can see that both metrics have quite low value across datasets for binary classification, while they are relatively larger for multi-class classification, which is because label model for binary classification is much easier to approximate.
Even for multi-class classification, all the EDs are still less then 22%, which indicates the approximated label model could replicate most of the predicted labels of original label model.
[Table] [Table]
G.5 How many mislabelings should be removed to have positive impact on test loss?
Here, we would like to answer the question of how many mislabeling to remove to have positive impact on
the
test loss.
Concretely, this could be reflected by a quantity β we call maximal removing portion (MRP).
That is, if we remove more than β % top-ranked negatively influential labelings, the resultant test loss after re-training will be larger than original test loss without any removing.
In other words, removing any portion of labelings between 0% and β % has a positive impact on the test loss.
We take label model MV and our RW method as an example for this study.
We found that the MRP β is closely correlated to the accuracy of training labels produced by label model (see Figure 3).
This is quite intuitive- for high-quality training set, the MRP would be low since we do not have to remove many labelings when training labels are already accurate, while for low-accuracy cases, the MRP would be high since most of the training labels are incorrect so we could remove a large portion of labelings but still be able to improve the test loss.
[Figure]
G.6 Additional experiments on improving test accuracy / F
In the main body of the paper
[Figure]
we leverage the proposed method to improve test loss; in this study
[Figure]
we instead present the improved model performance in terms of standard classification metrics,
i
[Figure]
e. ,
F1 score and accuracy.
Specifically, for binary classification, we adopt F1 score, while for multi-class classification, we use accuracy.
For simplicity, we do not include the methods based on RelatIF.
From the results, we can draw similar conclusion as Table 4 in the main body, i
[Figure]
e. , our methods outperform baselines in most of cases and achieve better averaged performance.
the classification metrics of end models
[Table]