Understanding Programmatic Weak Supervision via Source-aware Influence Function
Anonymous Author(s)
Abstract
Programmatic Weak Supervision (PWS) aggregates the source votes of multipleweak supervision sources into probabilistic training labels, which are in turn usedto train an end model.
With its increasing popularity, it is critical to have some toolfor users to understand the influence of each component ( e.g. , the source vote ortraining data) in the pipeline and interpret the end model behavior.
To achieve this,we build on Influence Function (IF) and propose source-aware IF, which leveragesthe generation process of the probabilistic labels to decompose the end model’straining objective and then calculate the influence associated with each (data,source, class) tuple.
These primitive influence score can then be used to estimatethe influence of individual component of PWS, such as source vote, supervisionsource, and training data.
On datasets of diverse domains, we demonstrate multipleuse cases: (1) interpreting incorrect predictions from multiple angles that revealsinsights for debugging the PWS pipeline, (2) identifying mislabeling of sources witha gain of 9%-37% over baselines, and (3) improving the end model’s generalizationperformance by removing harmful components in the training objective (13%-24%better than ordinary IF).
Introduction
One of the major bottlenecks for deploying modern machine learning models is the need for asubstantial amounts of human-labeled training data, often annotated over long periods of time andat great expense.
As machine learning models become increasingly powerful but also data hungry,new “data-centric” AI development workflows and systems have emerged, wherein the labeling anddevelopment of this training data is positioned and supported as the central development activity.
One recent and increasingly popular type of data-centric AI development uses Programmatic
Weak Supervision (PWS) , wherein users focus on developing a diversity of noisy, programmaticsupervision sources [32, 33, 46, 44] to programmatically annotate training data in an efficient way.
Specifically, these weak supervision sources , e g. , heuristics, knowledge bases, and pre-trainedmodels, are often abstracted as labeling functions (LFs) [33], which is a user-defined program thatprovides potentially noisy labels for some subset of the data.
So far, different modeling techniqueshave been developed to aggregate the noisy votes of LFs to produce training labels (often referred toas a label model )
Finally, these training labels are in turn used to train an end modelfor the downstream classification tasks.
In this study, we focus on this two-stage PWS pipeline [44].
With the increasing popularity of PWS in various applications [13, 36, 40, 19], it is of great importanceto provide practitioners with a tool that helps them understand the behavior of a trained end modelas an effect of each upstream component ( e.g. , source vote, training data, LFs, etc. )
involved inthe model’s training process.
First, by understanding what are the deciding factors that lead to amodel’s specific prediction, users are able to verify whether such influence is desirable in terms ofcritical aspects such as model safety and fairness [11, 35, 16].
Secondly, since developing LFs is still inevitably a partly manual process in real-world applications and may involve human expert in aniterative process [5, 20], it is beneficial to offer users with feedback of how individual component(like each LF) influences the end model performance on the downstream task.
Finally, the renderedunderstanding of (in)efficacy of the PWS pipeline can be naturally exploited to automatically improvethe end model performance.
Towards the goal of developing tools to help understand the model behavior, Influence Function(IF) [22, 15, 8] has been recently applied to interpret the model prediction as influence of the trainingdata for large-scale applications.
However, most of existing IF studies are unaware of the generationprocess of training labels like PWS and only estimate the influence of the training data [22, 3].
Notably, the group IF study [23] has attempted to apply IF on PWS, but it simply estimates theinfluence of each LF as summation of data influence and is limited to a specified PWS withoutlearnable label model.
In this work, we aim to develop a general framework that helps estimatethe influence of each PWS’s component on the end model and is aware of (yet computationallyagnostic to) the choice of label model.
To achieve this, we leverage the knowledge of how theprobabilistic labels are synthesized from sources and accordingly decompose the training loss ofeach data into multiple terms, each of which is associated with what we called ( i, j, c ) -effect , i.e. , theeffect of j -th LF on c -th dimension (corresponding to class c ) of the i -th data’s probabilistic label.
With such a training loss decomposition, we propose two means of calculating our source-aware IF ,namely, reweighting and weight-moving, based on different perturbations on the training respectively.
Each estimated source-aware influence score can be interpreted as the influence of removing the( i, j, c ) -effect from the pipeline.
By simply aggregating over specific dimension(s), it can be utilizedto estimate the influence of different components of PWS including (1) each source vote, (2) each LFand (3) each training data, since the influence score is additive [23].
In experiments, we exploit the source-aware IF in a diversity of 13 classification datasets includingtabular, text, and image data, and present multiple applications.
First, we show that source-aware IFcan help to understand the behavior of the end model in terms of the PWS components; in particular,when interpreting the same incorrect prediction made by two PWS pipelines with different labelmodels, data-level IF might lead to the same influential training data, whereas source-aware IF iscapable of revealing the most responsible LF or source vote.
Second, we use source-aware IF asa tool to identify mislabeling of each LF and show that it outperforms baselines with a significantmargin of 9%-37%.
Finally, we demonstrate that the training loss decomposition and source-aware IFenable fine-grained training loss perturbation and consequently lead to better test loss improvement(13%-24% improvement over ordinary
IF [22] and group IF [23]).
To summarize, this work makes the following contributions.
First , we propose source-aware IFtailored for PWS that incorporates our knowledge of the generation process of training labels.
Second , we develop two means of calculating the source-aware IF based on different training lossperturbations.
Third , we demonstrate a variety of use cases of the source-aware IF and conductextensive quantitative and qualitative experiments on real datasets in different domains to verify theefficacy of source-aware IF on understanding and improving PWS pipelines.
Preliminary
We denote scalars and generic items as lowercase letters, vectors as lowercase bold letters, andmatrices as bold uppercase letters.
We use superscripts to index generic items, e.g. , the i -th data x ( i ) ,and use subscripts to index certain dimensions of a vector, e.g. , the i -th component of a vector v i , orthe vector output of a function f ( · ) i .
All derivation details are deferred to the appendix.
Programmatic Weak Supervision.
We target at a C -way classification problem and have a trainingdataset of size N , i.e. , D
=
x ( i ) , y ( i ) }
i ∈
[ N ] ,
where x ( i ) ∈
X
= R d
and y ( i )
∈ Y =
[ C ] .
Theground truth y
( i ) s are not observed.
We have M labeling functions (LFs) { λ ( j ) ( · ) } j ∈ [ M ] .
Eachλ ( j ) ( · ) ∈ {− 1 } ∪ Y ( j ) , where Y ( j ) ⊂
Y .
That is, each λ ( j ) ( · ) either abstains ( − 1 ) or outputsa specific label
y ∈ Y ( j ) on a given data point.
We define the label matrix as L ij = λ ( j ) ( x ( i ) ) ,and with a slight abuse of notation, use L ( x ( i ) ) to represent the row of L corresponding to allof the labels assigned to x ( i ) .
The goal of Programmatic Weak Supervision (PWS) is to train anend classifier h θ : X → Y with { x ( i ) } i ∈
[ N ] and L only, where h θ ( x ) = arg max c ∈
[ C ] f θ ( x ) c andf θ ( x ) ∈ R C is a vector-valued function parameterized by θ .
A typical PWS pipeline involves a label model g W ( L ( x )) parametrized by W , which aims to produce a weak probabilistic labelˆ y ( x ( i ) )
=
g W ( L ( x ( i ) )) ∈ ∆ C for each data point x
( i ) (denoted as ˆ y ( i ) for simplicity).
The labelmodel produces a dataset ˆ D
= { x ( i ) , ˆ y ( i ) }
i ∈
[ N ] , which is then used to train f θ via the noise-awareloss [33] ˆ ℓ (ˆ y ( i ) , f θ ( x ( i ) ))
=
E y ∼ ˆ y
( i )
[ ℓ ( y, f θ ( x ( i ) )]
= (cid:80) Cc
=
ˆ y
( i ) c ℓ ( c, f θ ( x ( i ) )) , where ℓ ( · , · ) is thecross-entropy loss, leading to the following training objective:
[Equation]
where θ ⋆ is the minimizer of the loss ˆ L ( ˆ D ; θ ) over the training set ˆ D .
Influence Function.
The Influence Function (IF) estimate how the minimizer θ ⋆ of the empiricalloss ℓ would change if we were to reweight the i -th training example
z ( i ) =
( x ( i ) , y ( i ) ) by ϵ i .
The keyidea is to make a first-order Taylor approximation of the changes in θ ⋆ around ϵ i = 0 .
Specifically, ifthe i -th training sample is upweighted by a small ϵ i , the perturbed risk minimizer θ ⋆ϵ i becomes:
[Equation]
The changes of the model parameters due to the introduction of the weight ϵ i are:
[Equation]
where H θ ⋆ = 1 N (cid:80) i ′ ∇ 2 θ ℓ (cid:16) y ( i ′ ) , f θ ⋆ ( x ( i ′ ) ) (cid:17) is the Hessian of the objective at θ ⋆ and assumed to bepositive definite [22].
Similarly, the change of loss with respect to a single sample z ′ =
( x ′ , y ′ ) is
[Equation]
Then, as shown in previous work [24], the change of loss over a holdout set , e.g. , the validation set
D v , is simply the sum of the changes over every sample in the set:
[Equation]
The change of loss w.r.t.
one sample is often interpreted as its effect on model predictions over thetraining set [22], while the change of loss over a holdout set can be leveraged to improve the test lossby discarding/downweighting the training examples with negative impact [38, 42, 24].
Consideringthe extraordinary influence that might be caused by mislabeled examples, the relative influencefunction (RelatIF)
[3] has been proposed to measure the influence of a sample relative to its globaleffects.
The RelatIF can be defined as ϕ ′
i ( z ′ )
=
ϕ i ( z ′ )
√ ϕ
i ( z ( i ) ) .
Methodology
Given an unlabeled training set { x ( i ) } i ∈ [ N ] and M labeling functions (LFs) { λ ( j ) ( · ) } j ∈ [ M ] , we studythe scenario where a Programmatic Weak Supervision (PWS) pipeline is employed to synthesizeprobabilistic labels { ˆ y ( i ) } i ∈
[ N ] for training an end model f
θ ( x ) for a C -way classification problem.
We aim for a general framework for explaining and debugging the PWS pipeline through Influence
Function (IF)
[22], which estimates the influence of each training data on test loss/prediction for abetter understanding of model behavior and is able to improve/debug the training data accordingly.
In the PWS pipeline, we have full knowledge of how training labels are synthesized from differentsources, i.e. , LFs, which opens the opportunity to trace the influence in a source-aware way.
In thissection, we propose the source-aware IF tailored for PWS and describe its applications.
3.1 Training loss decomposition
By examining the popular choices of label models g W ( · ) [9, 33, 31], we show that their parameterscan be unifiedly represented as a tensor W ∈ R M × ( C +1)
× C , where the middle ( C + 1) dimensionsize is due to the additional "abstention" label LFs output.
The generation process of the probabilistic 125 labels can then be expressed as
[Equation]
where σ ( · ) could be (1) the identity function σ id ( x ) =
x for the classic Majority Voting label model 127 or its variants, or (2) the exponential function σ exp ( x ) = exp ( x ) to recover the softmax functionused in statistical models such as Dawid-Skene model [9] and Snorkel MeTaL [31].
Details can be 129 found in Appendix A.
In both cases, for the i -th data, a single parameter W j, L ij ,c controls the effectof the j -th LF’s vote L ij on the c -th dimension of the probabilistic label, i.e. , ˆ y ( i ) c .
For convenience,we denote the effect of L ij on ˆ y ( i ) c as the ( i, j, c ) -effect , as it uniquely corresponds to the i -th data,the j -th LF and c -th dimension of the aggregated probabilistic label.
With such unified formulation 133 of probabilistic label generation, we rewrite the noise-aware loss [33] as
[Equation]
We proceed with the σ ( · ) being the identity function because it enables the loss decomposition 135 introduced below.
We will later discuss other choices of σ ( · ) .
By simple rearrangement, we coulddecompose the loss over the training set into the summation of all N × M × C terms:
[Equation]
Such a decomposition is source-aware in the sense that it preserves how the source votes areaggregated via the label model, and each ¯ ℓ i,j,c ( θ ) indeed corresponds to the ( i, j, c ) -effect.
3.2 Source-aware Influence Function
As a consequence of training loss decomposition, we are now able to compute the influence score foreach term ¯ ℓ i,j,c ( θ ) , which is the influence of removing ¯ ℓ i,j,c ( θ ) or, in other words, the ( i, j, c ) -effectfrom training.
We provide two means of calculating the influence score: Reweighting and Weightmoving , which correspond to two different ways of perturbing the training loss.
Specifically, givenan (unseen) instance z ′ =
( x ′ , y ′ ) and the cross-entropy loss ℓ ( y ′ , f θ ( x ′ )) , we are interested in theinfluence score defined by the change of loss ℓ ( y ′ , f θ ( x ′ ))
caused by the differences in the modelparameters θ after perturbing the training loss ˆ L ( ˆ D ; θ ) .
We first give the formulation of an ordinary
IF for a training data z ( i ) =
( x ( i ) , ˆ y ( i ) ) in the context of PWS, which is smilar to Eq [Equation] [Equation]
[Equation]
Here, H − 1 θ ⋆ = 1 N (cid:80) Ni
=1 ∇ 2 θ ˆ ℓ (ˆ y
( i ) , f θ ⋆ ( x ( i ) ))
and θ ⋆ is the minimizer of the training loss before theperturbation.
We reuse the notation of H θ ⋆ and θ ⋆ in this section as their meanings are unchanged.
Reweighting.
To study the influence of removing the ( i, j, c ) -effect, we reweight ¯ ℓ i,j,c ( θ ) such that
[Equation]
Then, the influence score, i.e. , the difference of loss on z ′ =
( x ′ , y ′ ) caused by reweighting the lossterm ¯ ℓ i,j,c ( θ ) , is,
[Equation]
In contrast to ordinary IF that reweights the loss w.r.t.
individual training data, the proposed one isfiner-grained since it reweights the decomposed loss terms.
By reweighting, the resultant probabilistictraining label is no longer sum-to-one and the solution (Eq. 11) is only for identity σ ( · ) function.
[Table]
Weight-moving.
We also explore an alternative way of perturbing the training loss called weight- 157 moving.
Different from reweighting, its computation is agnostic to the σ ( · ) function and the probabilistic training label is still sum-to-one.
Specifically, we remove the ( i, j, c ) -effect in the probabilistictraining label ˆ y ( i ) and renormalize it, leading to a new probabilistic training label:
[Equation] [Equation]
[Table]
[Equation]
[Table] [Equation]
[Equation]
Then, we perturb the training loss by moving the weight of the original loss on the i -th dataˆ ℓ (ˆ y ( i ) , f θ ( x ( i ) )) to a new loss term ˆ ℓ (ˆ y
( i ) − j ′ c ′ , f θ ( x ( i ) )) :
[Table]
[Equation]
[Table]
[Equation]
When ϵ i,j,c = − 1 N , the loss term ˆ ℓ (ˆ y
( i ) , f θ ( x ( i ) )) is indeed replaced by the new lossˆ ℓ (ˆ y ( i ) − j ′ c ′ , f θ ( x ( i ) )) .
We again provide the formulation of corresponding influence score:
[Equation]
Discussion.
Note that the change of loss over a holdout set D v can be similarly computed andwe denote it as ¯ ϕ rwi,j,c
( D v ) and ¯ ϕ wmi,j,c
( D v ) .
We omit the input variable and the superscript and use¯ ϕ
i,j,c to represent the proposed source-aware IF in general.
The ¯ ϕ i,j,c corresponds to the influenceof a specific perturbation on the training loss: setting the label model parameter W j, L ij ,c to zerofor
the i
-the data so that the ( i, j, c ) -effect is removed from training.
Then, the difference betweenreweighting and weight-moving reduces to whether renomarlization on the probabilistic labels isperformed after the perturbation.
Extension to RelatIF [3].
We additionally demonstrate how to compute the RelatIF [3] in oursource-aware IF setting, Specifically, the self-influence is ¯ ϕ selfi,j, [Equation] =
[Equation]
H − 1 θ ⋆ ∇ θ ¯ ℓ i,j,c ( θ ⋆ ) .
Then, the source-aware RelatIF for both ¯ ϕ rwi,j,c and ¯ ϕ wmi,j,c are ¯ ϕ rwi,j,c (cid:113) ¯ ϕ selfi ,j,c and ¯ ϕ wmi,j,c (cid:113) ¯ ϕ selfi ,j,c , respectively.
3.3 Use
Case 1: estimating influence of different components.
One major benefit of our training lossdecomposition and the identity σ ( · ) function is that we could readily reuse the source-aware IF ¯ ϕ
i,j,cto estimate different components of the PWS pipeline, e.g. , individual data or LF.
This is becauseaccording to the study of group influence [23], when measuring the change in test prediction/loss,the influence is additive, i.e. , the influence of a set is the sum of influences of its constituent points.
However, such a nice property only holds true for the reweighting method, while, in theory, theinfluence calculated by weight-moving does not follow due to the fact that they are not additive.
We list the influence of different components of a PWS pipeline as well as how to calculate themusing the fine-grain
IF ¯ ϕ i,j,c in Table 1.
Note that the 1st and 2nd row of Table 1 indeed recover theordinary IF of a training data [22] and a LF [23], respectively.
Case 2: improving test loss.
Another application of IF is to improve the test loss by identifying andthen discarding/downweighting the harmful training data [24, 42, 38].
Similarly, the source-aware IF¯ ϕ i,j,c can be employed to improve the test loss by discarding/downweighting the loss term ¯ ℓ i,j,c ( θ )with negative impact.
Specifically, we have the following theorem for reweighting (a similar resultfor weight-moving can be found in Appendix E.2):
Theorem 1.
Discarding or downweighting the loss terms in S − = { ¯ ℓ i,j,c ( · ) | i ∈
[ N ] , j ∈ [ M ] , c ∈
[ C ] , ¯ ϕ rwi,j,c ( D t ) > 0 } from training could lead to a model with lower loss over a holdout set D t :
[Equation]
where θ ⋆ S − is the optimal model parameters obtained after the perturbation.
In practice, we use the validation set D v and S − ( α ) =
{ ¯ ℓ i,j,c ( · ) | i ∈
[ N ] , j ∈ [ M ] , c ∈
[ C ] , ¯ ϕ rwi,j,c
( D v ) > α } to tolerate the estimation error of IF where the hyperparameter α controlsthe proportion of items to be perturbed [24].
Note that this method of improving test loss relies onthe training loss decomposition and σ ( · ) being the identity function.
3.4 Beyond the identity function
The aforementioned use cases rely on the additivity of influence score and the training loss decompo- 199 sition, which are both consequences of σ ( · ) being the identity function.
When σ ( · ) is the exponentialfunction, we could approximate the label model g W ( · ) by a new label model ¯ g ¯ W ( · ) with parameter¯ W of the same shape but using identity function instead.
Obviously, we want the new label model¯ g ¯ W ( · ) to reproduce the same probabilistic labels as those generated by g W ( · ) .
This motivates us tosolve the following optimization problem for ¯ W :
[Equation]
[Table]
Then, we can replace the label model g W ( · ) with ¯ g ¯ W ⋆ ( · ) everywhere.
As a consequence, we arenow able to unify various types of label models in a single framework and enjoy the convenience andadvantages brought by the linearity of the identity function.
Note that for the original label modelg W ( · ) with σ ( · ) being the exponential function, we can still derive the influence
¯ ϕ
i,j,c (see appendix).
We study the effect of replacing original label model with the approximated one on the end modelperformance in the experiment section.
Experiment
4.1 Setup
Datasets.
We include the following four classification datasets in WRENCH [46], a collection ofbenchmarking datasets for PWS: Census , Youtube , Yelp , and IMDb .
Note that the Census dataset isa tabular dataset while the others are textual datasets.
We use the labeling functions (LFs) released by
WRENCH [46].
We also include the following tabular datasets: Mushroom [12], Spambase [12], and
PhishingWebsites (PW) [29], for which we follow the instruction in the WRENCH [46] codebase 1to generate LFs from a decision tree learned on the labeled data.
Finally, we follow [28] to derive
LFs for a multiclass image classification task using the DomainNet [30] dataset, which contains 345classes of images in 6 different domains: real, sketch, quickdraw, painting, infograph and clipart.
We construct a classification task for each domain with the 5 classes containing the largest numberof instances.
We then train classifiers using the selected classes within the remaining five domainsas LFs.
We name the 6 resultant datasets DN-real , DN-sketch , DN-quickdraw , DN-painting ,
DN-infograph , and DN-clipart for each domain, respectively.
Following [46], we use a pre-trained
BERT model [10] to extract features for textual data and use ResNet-18 [17] pre-trained on ImageNetfor image data.
Model Training and Evaluation.
For label model, we focus on three commonly-used choices:
Majority Voting (MV), Dawid-Skene model (DS) [9], and Snorkel [31].
The σ ( · ) function of the lattertwo label models is the exponential function and we use their approximated variants (as mentioned in
Section 3.4) for all experiments.
Following [23], we use logistic regression for the end model anddefer the experiments on using neural networks to the appendix.
We train the end model initializedas zero by gradient descent for 10,000 epochs with learning rate being 0.001, and then calculate
[Table]
the influence score on validation set D v throughout the experiments.
When using the test loss asevaluation metric, we use D v to do model selection and report the test loss of the select model.
For simplicity, we use IF (R-IF) to represent ordinary IF (RelatIF) baseline; RW (R-RW) and
WM (R-WM) are the proposed source-aware IF (RelatIF) calculated by the reweighting and weightmoving method respectively.
We also use the term SIF
( R-SIF ) to represent the source-aware IF(RelatIF) when we do not care whether it is calculated by reweighting or weight-moving.
Additionalexperiments and details are in Appendix G.
4.2 Experimental results
Understand model behavior from multiple angles.
First and foremost, source-aware IF can beused to understand the end model behavior from multiple angles, while ordinary IF can only identifyinfluential training data.
Here, we take the DN-real dataset as an example and use reweighting-basedsource-aware IF ( RW ) to diagnose misclassified test data.
For a single misclassified test data of theend model, we present the most "responsible" (1) training data, (2) LF, and (3) labeling (the vote of a
LF on a training data), where being the most "responsible" means having the highest influence scoreon that data and likely being the primary cause of the misclassification.
In Table 2, we show that forthe same image misclassified by the end model trained with different label models, we could havetotally different explanations and components of the PWS pipeline to blame on.
Specifically, for a“ dog ” image misclassified as “ golf club ”, we can see that for both the DS and Snorkel label models,the most "responsible" training image are the same but the most "responsible" LF and labeling aredifferent: for DS the LF trained with images of painting domain is likely the primary cause for themisclassification, while for Snorkel we might blame more on the LF trained with clipart images.
Without source-aware IF, we can hardly distinguish the cause of misclassification for DS and Snorkelbased solely on the training data as the most "responsible" one is identical.
Identify mislabeling of LFs.
One unique advantage of source-aware IF over ordinary IF is that itallows us to trace the influence of each LF’s labeling on a specific training data ( ϕ L ij =
(cid:80) Cc =
1 ¯ ϕ i,j,c ,the 3rd row in Table 1), which is helpful in understanding the expertise of each individual LF andthus provides clues for practitioners to debug LFs, while ordinary IF can only estimate the influenceof training data as a whole.
We use ϕ L ij as a scoring function of how likely the j -th LF mislabelsi -th data, i.e. , L ij ̸ =
y ( i ) , to identify the mislabeling of individual LF.
We thus formulate it as abinary classification task and report the average precision (AP), a standard evaluation metric ofa scoring function.
We use the discrepancy between j -th LF’s labeling on i -th data ( L ij ) and theprobabilistic label output by (1) label model ( LM ), (2) end model ( EM ), and (3) a K -nearest neighborclassifier ( K = 10 ) trained on validation set ( KNN ) as baseline scoring functions.
Specifically, wecalculated the discrepancy as one minus the dot product of L ij ’s one-hot representation and theprobabilistic label; for example, if L ij ’s one-hot representation is [0 , 1 , 0] ⊤ and the probabilisticlabel is [0 . 5 , 0 . 4 , 0 .
3] ⊤ , the discrepancy is 1 − 1 × 0 .
4 = 0 .
Note that KNN is a more fair andcompetitive baseline because it directly leverages the validation set and, as our source-aware IF, doesnot introduce any parametrized model in addition to what PWS has.
We present the results in Table 3;we can see the superiority of source-aware IF in this task since it outperformed baselines in mostcases and the averaged gains are about 9%-37% for different label models.
Intriguingly, when LMand EM render unsatisfactory performance, e.g. , on DomainNet , which indicates the quality ofthe label or end model is not good, the source-aware IF still performed well.
This means it is lesssensitive to the quality of PWS, which is critical for debugging the pipeline.
[Table]
[Figure] [Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
Improve test loss.
We compared the proposed source-aware IF as well as its RelatIF variant toordinary IF
RelatIF in terms of the capability of improving the test loss.
For source-aware IF/RelatIF,we discarded loss terms in S − ( α ) and then re-trained the end model (as described in Section 3.3),while for IF/RelatIF, we likewise discarded training data in D − ( α ) =
{ z ( i ) | i ∈
[ N ] , ϕ
i ( D v ) > α } .
For these methods, we tuned the hyperparameter α on validation set for best validation loss.
Weadditionally included the group influence ( G-IF ) [23] that estimates the influence of each LF andis only applicable to MV as in [23]; we discarded k LFs with highest negative impact and tune kon validation set.
The results are in Table 4, where ERM (Emperical Risk Minimization) is theordinary training without any perturbation.
From the results, we conclude that although ordinary
IF/RelatIF can already improve the test loss over ordinary training ( ERM ) by a large margin (>0.1test loss on average), the source-aware IF/RelatIF can further boost the performance by a similarmargin (>0.1 averaged test loss improvement over IF/RelatIF).
This shows the benefit of employing
IF in a source-aware manner and provides insights for practitioners to improve the PWS pipeline.
Discussion.
One may develop sophisticated methods to identify mislabeling of LFs or to improvetest loss and achieve better performance than source-aware IF, but source-aware IF is designed as atool of interpreting the end model prediction and when used in these applications, it does not introduceadditional parameterized model.
Also, we believe that source-aware IF can inspire advanced methodsin the future and be leveraged by them to further boost the performance on the task of identifyingmislabeling of LFs or improving test loss.
4.3 Additional studies
Estimate ordinary data-level IF via source-aware IF.
As described in Section 3.3, we can usethe source-aware IF calculated by the reweighting method to estimate the data-level influence score( ϕ x ( i ) =
(cid:80) Mj
=1 (cid:80)
Cc =1 ¯ ϕ
i,j,c , the 1st row in Table 1), which coincides with the influence computedby ordinary IF in theory.
Thus, we are curious about how well the source-aware IF can be used toestimate the ordinary data-level IF.
Because IF is known to be noisy in the value but good in theirranking [4, 23], we instead study the ranking correlation between ϕ x ( i ) and the ordinary IF.
In Table 5 we report the Spearman’s ranking correlation coefficient ( ≤ 1 ), where larger coefficients indicatebetter estimation of the ordinary IF based on ϕ x
( i ) in terms of the ranking.
From the results, we cansee that the ϕ x ( i ) actually preserves the ranking of ordinary IF quite well as Spearman’s coefficientsare from 0.71 to 0.99, which further demonstrates the efficacy of source-aware IF. [Figure]
[Figure]
Efficacy of label model approximation.
Since we bypassedthe non-linearity of exponential function in DS and Snorkelwith their approximated variants (Section 3.4), it is natural toask how well the approximated label model can perform andwhether or not the approximation is well-haved.
To answerthese questions, in Figure 1 we show that although the averaged test loss of ordinary training ( ERM ) and best of IF/R-IF
( Best IF/R-IF ) are slightly better than their counterpart ofapproximated label model ( Approx.
ERM and Approx.
Best
IF/R-IF ), the best averaged test loss obtained by SIF/R-SIFwith the approximated label model ( Approx.
Best SIF/R-SIF )is much lower.
Such an observation indicates that with the goalof improving test loss, it is beneficial to choose σ ( · ) to be theidentity function so that the training loss can be decomposedand perturbed based on source-aware IF.
Related work
Programmatic Weak Supervision.
In the Programmatic Weak Supervision (PWS) literature [33,44], popular choices of labeling functions (LFs) include user-crafted heuristics [32, 39], pretrainedmodels [2, 28], external knowledge bases [18, 26], and crowd-sourced labels [9, 25].
Recently,researchers have developed various label models [33, 31, 14, 40, 45, 43, 37].
In the PWS pipeline, thebehavior of the end model is directly dictated by the soft labels, wherein the soft labels themselves arealso intermediate products of the label model and the LFs’ votes.
Thus, to interpret the trained endmodel and potentially debug the PWS workflow, it is critical to understand the influence of each PWScomponent on the end model’s performance, which has yet received adequate research attention.
Influence Function.
Among various ways to understand a model’s behavior through the trainingdata points, Influence Function (IF) is an effective technique with a wide range of applications [22,23 , 16, 7].
In PWS, although one can directly apply IF to understand a model’s behavior through theweakly labeled training data, more insights may be unveiled by further investigations of how each
PWS component, e.g., LFs and label models, influences the end model.
In similar spirit to us, [6]developed multi-stage IF to trace a fine-tuned model’s behavior back to the pretraining data.
Perhapsmost similarly to our work, [23] conducted initial exploration on using group influence to study howremoving an LF from the PWS pipeline would affect the end model’s performance.
However, thegroup influence method is only applicable to a specific simplified setting of PWS.
Conclusion
In this work, by leveraging the knowledge of how probabilistic training labels are aggregated fromdifferent sources in the PWS pipeline, we have proposed source-aware IF that enables users toestimate the influence of different components ( e.g. , source vote, training data, LFs, etc. .)
in thepipeline on the end model’s behavior, and unlocked various practical use cases.
Specifically, we havedemonstrated how the proposed source-aware IF can be used to (1) unveil the most influential trainingdata point, LF, or a specific vote to a misclassification made by a model, (2) identify mislabeling of
LFs on certain data points, and (3) improve the end model’s test performance.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information onhow to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or[N/A] . You are strongly encouraged to include a justification to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description. For example: • Did you include the license to the code and datasets?
[Yes] See Section.
• Did you include the license to the code and datasets?
[No] The code and the data are proprietary.
• Did you include the license to the code and datasets? [N/A] Please do not modify the questions and only use the provided macros for your answers. Note that the
Checklist section does not count towards the page limit. In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below. For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See appendix.(c)
Did you discuss any potential negative societal impacts of your work?
[Yes] Seeappendix.(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] See appendix.(b)
Did you include complete proofs of all theoretical results?
[Yes] See appendix.3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)?
[Yes] See supplemental materials.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[No] For experiments with randomness, we report averagedresults over 5 runs. [Table]
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See appendix.
[Table]
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
[Table]
(b) Did you mention the license of the assets?
[Yes]
[Table] (c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
[Table] (d) Did you discuss whether and how consent was obtained from people whose data you’re [Table]
using/curating?
[Yes]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[Yes]
[Table] If you used crowdsourcing or conducted research with human subjects...
[Table]
(a) Did you include the full text of instructions given to participants and screenshots, if
[Table]
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
[Table]
[-we have exp( x ) = 
-]
Board (IRB) approvals, if applicable?
[N/A]
[Table]
(c) Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?
[N/A]
[Table]