A CTIVE R EFINEMENT OF W EAKLY S UPERVISED M ODELS
Anonymous authors
Paper under double-blind review
A BSTRACT
Supervised machine learning (ML) has fueled major advances in several domains such as health, education and governance.
However, most modern ML methods rely on vast quantities of point-by-point hand-labeled training data.
In domains such as clinical research, where data collection and its careful characterization is particularly expensive and tedious, this reliance on pointillistically labeled data is one of the biggest roadblocks to the adoption of modern data-hungry ML algorithms.
Data programming, a framework for learning from weak supervision, attempts to overcome this bottleneck by generating probabilistic training labels from simple yet imperfect heuristics (or labelling functions) obtained a priori from domain experts.
We present WARM , Active Reﬁnement of Weakly Supervised Models, a principled approach to iterative and interactive improvement of weakly supervised models via active learning.
WARM directs domain experts’ attention on a few selected data points that, when annotated, would most improve the label model’s probabilistic accuracy.
Gradient updates are then backpropagated to iteratively update the parameters of the individual expert labelling functions in the weak supervision model.
Experiments on multiple real-world medical classiﬁcation datasets reveal that WARM can substantially improve the accuracy of probabilistic labels used to train downstream classiﬁers, with as few as 30 queries to experts.
Additional experiments with domain shift and artiﬁcial noise demonstrate WARM ’s ability to adapt to changing population characteristics as well as noisy initial labelling functions from the experts.
These capabilities make WARM a potentially useful tool for deploying, maintaining, and auditing weakly supervised systems in practice.
I NTRODUCTION
Machine learning (ML) has seen widespread adoption in several domains such as health, education and public policy.
In the clinical context, supervised ML algorithms have been shown to be effective solutions to a gamut of problems, ranging from detecting abnormal heart rhythms in electrocardiogram (Goswami et al., 2022; Hannun et al., 2019), prognosticating neurological recovery using continuous electroencephalogram (Elmer et al., 2016; 2020) to detecting lung cancer in CT scans (Gao et al., 2019).
However, most modern ML algorithms, especially large deep neural networks, rely on vast quantities of pointillistically labeled training data.
In our experience, while raw clinical data is abundant, its careful annotation is not.
Manually labeling large quantities of clinical data is often tedious, prohibitively expensive, prone to error, and acutely unscalable.
Recently, many studies have explored the use of cheap, potentially noisy sources of supervision to noisily annotate large training sets (Goswami et al., 2022; Fries et al., 2019; Saab et al., 2020) using weak supervision.
These methods combine these sources to create noisy labels on unseen data, which can then be used to learn powerful discriminative end models.
In principle, these methods resemble other repeated-labeling techniques such as crowd-sourcing (Li et al., 2013), or distant supervision relying on external knowledge bases (De Sa et al., 2016).
In this paper, we leverage data programming (Ratner et al., 2016) which uses a generative model to integrate expert knowledge expressed in the form of noisy heuristics ( labeling functions or LFs) to probabilistically label training data.
Since the quality of the training labels is strongly correlated with that of labeling functions, unreﬁned weak supervision may not always yield optimal performance.
Hence, in practice, experts using data programming often spend several hours iteratively designing new heuristics or reﬁning existing ones (Varma & R´e, 2018).
Our work aims to streamline and reduce this effort.
Another closely related issue is that of domain shift (also referred to as knowledge shift ).
Imbalance in characteristics of source and target populations may render ML models trained on one domain perform poorly on another.
This domain shift, especially in the form of population characteristics like gender or race, is prevalent in clinical settings (Thiagarajan et al., 2018).
In the context of weak supervision, we are interested in the related problem of knowledge shift.
Speciﬁcally, we want to train effective ML classiﬁers while tailoring knowledge of the source population to the target population (e.g. heart disease in young and old patients) via small changes in the LFs.
To this end, we propose WARM (Active Reﬁnement of Weakly Supervised Models), a principled approach towards quick, iterative, and interactive reﬁnement of weakly supervised models 1 .
WARM directs expert attention to speciﬁc data points in the training set, which if labeled, would improve the labeling functions and the resulting label model the most.
Speciﬁcally, at each iteration WARM directs an expert to label the most uncertain data point, and then updates decision parameters of the labeling functions to reduce the uncertainty.
Experiments on multiple real-world medical classiﬁcation datasets of varying sizes and complexity, reveal that with as few as 30 queries to an expert, WARM can signiﬁcantly improve label model accuracy resulting in higher quality training labels.
Moreover, WARM outperforms existing approaches to active weak supervision (Biegel et al., 2021; Nashaat et al., 2018) and supports interpretability by allowing experts to track changes in LFs.
Moreover, auxiliary experiments on a subset of our datasets show that WARM can not only combat knowledge shift but also de-noise LFs.
For example, we found that WARM could adapt LFs to detect the presence of heart disease deﬁned on a young population of patients, to an older population, resulting in a signiﬁcant 1 − 3% increase in label and end model performance.
We also found that WARM was able to easily reﬁne noisy LFs to achieve label and end model performance on par with no noise.
R ELATED W ORK
There is a small but growing body of work on active strategies for weak supervision.
A few studies combine data programming with active learning on data points.
For instance, Nashaat et al.
(2018) proposed a method which actively obtained expert-labeled data via uncertainty sampling and corrected LF votes on individual data points to reﬂect the true labels.
While their method improved the probabilistic predictions, it did not change the label model per se.
Biegel et al.
(2021) recently proposed Active WeaSuL which uses experts labels to learn how to best combine LFs.
Speciﬁcally, they add a penalty term to the data programming loss function to nudge its optimal parameters to a conﬁguration where the label model predictions agree with the expert labels.
They also proposed a novel active sampling strategy tailored to the problem, which splits the data into buckets and measures the KL-divergence between the expert and estimated probabilistic labels to query new labels.
Some other studies have instead used labeled data to inspire ideas for new labeling functions (CohenWang et al., 2019), and yet others have used them to automatically come up with new one (Awasthi et al., 2020;
Boecking et al., 2020).
Cohen-Wang et al.
(2019) for instance proposed two simple strategies ( i.e. data points where LFs abstain or disagree the most) to iteratively select subsets of data to inspire domain experts to create new LFs.
On the other hand, Boecking et al.
(2020) introduced an interactive weak supervision method that automatically improves a set of LFs based on feedback from users.
Alternatively, to reduce human effort to an absolute minimum, some studies have attempted to automate weak supervision by automatically generating heuristics based on a small labeled subset of the data (Varma & R´e, 2018; Nashaat et al., 2020).
For instance, Varma & R´e (2018) proposed Snuba which automatically trains decision trees, logistic regression models and k-nearest neighbor classiﬁers as LFs, tuned for accuracy and coverage (amount of data that the LF predicts a class as oppose to abstaining from predicting any class), and later pruned to ensure sufﬁcient dissimilarity.
Finally,
Nashaat et al.
(2020) proposed Asterisk , extending their prior work on active weak supervision by now automatically generating heuristics from a small labeled dataset.
In our experience, these methods are highly dependent on the quality and size of the labeled validation set.
Furthermore, they may require large diverse labeled datasets for complex prediction tasks as evidenced by our experiments on the diabetes dataset .
Additionally, Asterisk needs a large amount (up to ∼ 7% of training data) of actively labeled data points for good performance.
M ETHODOLOGY 3. P ROBLEM F ORMULATION AND M ODELING A SSUMPTIONS
Let ( x , y ) ∼ D denote the data generating distribution, where each data point x ∈ R d is associated with a latent class label y ∈ { 1 , . . .
, k } .
Users provide an unlabeled training dataset
S u
= { ( x i , y i ) } ni
=1 , where y i is unobserved, and m soft unipolar labeling functions (LFs) Γ which we deﬁne as follows.
Since we only consider LFs that predict a single class, let pol ( f ) ∈ { 1 , . .
. , k } deﬁne the polarity (or the single class that it predicts) of a LF f .
We deﬁne Γ = { γ τ,j } mj =1 , γ τ,j : R d −→ [0 , 1] , such that γ τ,j ( x i , c ) (cid:44) 1 { c = pol ( γ τ,j ) }
ˆ p ( y i =
c
| x i ) , where ˆ p is any arbitrary scoring function differentiable with respect to its parameters τ .
Speciﬁcally, each LF γ τ,j depends on one or more parameters in the parameter vector τ and expresses the likelihood that a data point x i belongs to a particular class as identiﬁed by the polarity function pol ( γ τ,j ) .
We drop the subscript τ in the rest of our discussion for the sake of brevity.
3. M OST C OMMON C LASSES OF L ABELING F UNCTIONS ARE D IFFERENTIABLE
[Equation]
ii) soft equivalents of a labeling function to characterize normal clinical ﬁndings in EEG data.
While both versions make identical predictions, the latter is differentiable with respect to its decision parameter,
high baseline .
Fig.
1(ii) is an example of an expert-deﬁned LF to characterize normal clinical ﬁndings in electroencephalogram (EEG) recordings 2 which depends on the high baseline parameter.
Our methods assume that all the LFs in the label model are differentiable with respect to their decision parameters.
Many frequently used classes of LFs are intrinsically differentiable with respect to their parameters, such as multi-layer perceptrons, convolutional neural networks, log-linear models, neural language models (Hooper et al., 2020; Varma & R ´ e, 2018).
Other common classes of LFs can be easily transformed into their soft counterparts.
One example is decision stumps ( IF <CONDITION>
THEN
CLASS <I> ELSE CLASS <J> ) (Varma &
R´e, 2018).
More generally, a AND or OR could also be made differentiable by fuzzy logic.
To transform crisp decision stumps into their soft equivalents, we can follow prior work on decision trees (Su´arez & Lutsko, 1999).
Specifically, consider a heuristic of the form: λ (cid:44) d (cid:124) x > a , where d (cid:124) x deﬁnes a new composite variable and a ∈ τ is a decision parameter.
Then we deﬁne its fuzzy equivalent as γ (cid:44) sigmoid ( d (cid:124) x − a ) , where sigmoid ( x ) =
1+exp( − x ) .
Intuitively, while λ indicates which side of the hyperplane d (cid:124) x
= a a point lies, γ encapsulates a probabilistic notion of distance from the decision boundary.
Using the corresponding 2 Speciﬁcally, the LF is developed to work with amplitude-integrated EEG, which is a widely used quantitative summary of raw multi-channel EEG data.
Zadeh operators, conjunctions are converted via ( AND ( x, y ) → MIN ( x, y ) ), disjunctions are converted via ( OR ( x, y )
→ MAX ( x, y ) ) and negations are converted via ( NOT ( x ) → (1 − x ) )
(Zadeh, 1996).
There are still other ways to obtain soft equivalents of crisp LFs, for instance by training feed-forward neural networks to approximate classes of functions under reasonable assumptions on their continuity and topology.
This extension of crisp LFs into their soft counterparts not only enables them to express conﬁdence on their predictions, but also better capture fuzzy decision boundaries, common in most practical settings.
Additionally, restricting LFs to be unipolar can be done without any loss of generality, since complex LFs can be easily broken down into their unipolar components.
Moreover, in our experience, several simple unipolar LFs tend to perform better than a few complex multipolar LFs in practice.
Each soft LF γ j can be converted to its hard equivalent λ
j : R d −→ { 0 , pol ( λ j ) } using a threshold function, where 0 means that the LF abstained from voting for class pol ( λ j ) .
In particular, we have that
[Equation]
3. P ROBABILISTIC L ABEL E STIMATION
Given the unlabeled training dataset S u and an initial set of LFs denoted by Γ (1) , our goal is to efﬁciently improve the accuracy of LFs by reﬁning their parameters τ based on expert feedback, and ultimately assign probabilistic labels ˆ p ( y | Γ ( t ) , x ) to the training data.
Here Γ ( t ) denotes the set of LFs after t th iteration of WARM .
The probabilistic labels are used to further train a downstream classiﬁer f : X −→ Y .
We reﬁne the LFs automatically by asking domain experts to provide labels for a few highly uncertain data points, and back propagating LF errors to update their parameters τ .
In order to estimate probabilistic labels p ( y | Γ ( t ) , x ) , we propose the following weighted majorityvote label model:
[Equation]
such that ˆ p il = p ( y i = l | Γ ( t ) , x i ) is the estimated probability that x i belongs to class l , while b l = p ( y = l ) denotes its prevalence.
γ ( t ) j
( x i ) is the probabilistic vote of j th LF and ˆ θ ( t )
j is its unobserved accuracy, estimated using the data programming (DP) label model as deﬁned in Eq.(Ratner et al., 2016).
We ensure that ˆ p ik is a probability by passing ˆ p i through a softmax function, and note that our model is differentiable with respect to all LF parameters τ .
The DP label model is not differentiable with respect to LF parameters, and therefore cannot be directly used to estimate probabilistic labels while still having the ability to update LF parameters.
However, data programming has been shown to perform well in several weak supervision applications and has some theoretical guarantees for estimating the latent accuracy of LFs by optionally modeling dependencies between them.
Thus, in order to iteratively improve the accuracy of LFs and estimate probabilistic labels, we alternate between two objectives at the t th time step:
Estimate the accuracy of LFs keeping their parameters τ ﬁxed.
Let [Λ ( t ) ] n × m = { 0 , 1 , . .
. , k } n × m denote the observed matrix of crisp LF outputs at the t th time step, such that [Λ ( t ) ] ij = λ ( t )
j ( x i ) is the thresholded output of LF γ j on x i .
We deﬁne factors for LF accuracy and propensity as φ Acc ([Λ ( t ) ] ij , y i ) (cid:44)
{ [Λ ( t ) ] ij = y i } and φ Lab ([Λ ( t ) ] ij , y i ) (cid:44) 1 { [Λ ( t ) ] ij (cid:54) =0 } , respectively.
We then follow Ratner et al.
(2016) to deﬁne the joint distribution of the latent class label y and [Λ ( t ) ] as:
[Equation]
We use Snorkel (Ratner et al., 2017) to learn θ by minimizing the negative log marginal likelihood given the observed [Λ ( t ) ] :
[Equation]
We then set the accuracy weights of the LFs to be equal to ˆ θ ( t ) j .
Update LF parameters τ keeping their accuracy weights ﬁxed.
In this step, we use our label model (Eq. 1) to probabilistically label the training dataset, gather expert labels for the most uncertain data points, and ﬁnally update the LF parameters by minimizing the cross-entropy between the observed expert labels and probabilistic label predictions of the data points.
In particular, we update the LF parameters τ via gradient back propagation, so as to minimize the negative log likelihood of all the expert labels observed so far:
[Equation]
where S l is the set of expert labeled data points.
While in our experiments, at each time step t , we gather an expert label for the data point with the highest entropy, our methods are generic and can seamlessly incorporate other active learning acquisition functions.
Our algorithm is brieﬂy summarised in Algorithm 1.
Algorithm
1: procedure WARM ( [Γ (1) ] , S u
= { ( x i , y i ) } ni =1 , b ) 2: for t ∈ 1 to T do 3:
[Λ ( t ) ]
← c ([Γ ( t ) ]) (cid:46) Convert to crisp votes 4: ˆ θ ( t ) ← arg min θ − log (cid:18) (cid:80)
Y ∈Y p θ ( Y, [Λ ( t ) ]) (cid:19) 5: for i ← 1 to | S u | do 6: ˆ p il ← b l + (cid:80) mj
=1 γ ( t ) j
( x i , c )ˆ θ ( t ) j , ∀ l ∈ { 1 , . . .
, k }
(cid:46) Aggregate weighted LF votes 7: ˆ p i ← softmax(ˆ p i ) 8:
E i ← − (cid:80) kl
=1 ˆ p il log(ˆ p il ) (cid:46) Compute entropies 9: end for 10: y q ← query user ( { x q | q ← arg max i ∈ S u
E } )
(cid:46)
Query user for label 11:
S l ← S l ∪ ( x q , y q ); S u ← S u / ( x q , y q ) 12:
L CE ← (cid:80) ( x i ,y i )
∈
S l (cid:80) kl =1 − 1 { l = y i } log ˆ p il 13: Γ ( t +1) ← backprop errors (Γ ( t ) , L CE )
(cid:46) Update LF parameters 14: end for 15: end procedure
E XPERIMENTS 4. D ATASETS AND W EAK S UPERVISION S OURCES
Like most prior work on active and automated weak supervision, we primarily focus on the binary classiﬁcation setting for consistency (Varma & R´e, 2018; Biegel et al., 2021; Nashaat et al., 2018).
Speciﬁcally, we carry out experiments on various real-world clinical classiﬁcation datasets of varying complexity and sizes.
See Table 1 for a summary of all the datasets used.
All of the above datasets were taken from the UCI Machine Learning Repository (Dua & Graff, 2017).
Refer
Appendix A.1 for dataset details.
Since we did not have the requisite clinical expertise to deﬁne good weak supervision sources, we trained a Random Forest (RF) classiﬁer of decision stumps (depth-one decision trees) to automatically to simulate expert heuristics for simple LFs.
Speciﬁcally, we trained a Random Forest
Dataset
Classiﬁcation task
[Table]
Wisconsin Breast Cancer
Benign / malignant cancer     34 Wisconsin Diagnostic Breast Cancer
Benign / malignant cancer     37 Heart Disease
Absence / Presence of heart disease     55 Diabetes
Patient was
[Table]
-admitted or not 81412   46 EEG
Clinical ﬁndings in EEG     (37, 25, 27, 11)
[Table]
clusters      classiﬁer with 5 or 10 decision stumps on a 50% sub-sample of the training set, such that each decision stump served as a LF.
We used the Random Forest Classiﬁer implementation from the
Scikit-Learn (Pedregosa et al., 2011)
Python library for the same.
For all datasets, we acquired a set of 10 LFs, with the exception of the diabetes dataset for which we used a set of 5 LFs only.
Appendix A.6 contains examples of LFs for the Heart Disease dataset.
As in Biegel et al.
, we also benchmark WARM on a synthetic dataset comprising of two classes modeled by 2-dimensional isotropic Gaussians, with a training and testing set of 10,000 and 3, data points, respectively.
We use the same set of 3 LFs as Biegel et al.
(2021) for our experiments.
Finally, to subject WARM to even more realistic settings, we used it to iteratively reﬁne expert-deﬁned heuristics to classify clinical ﬁndings in electroencephalogram (EEG) recordings.
The dataset comprised of continuous 6-hour amplitude-integrated EEG (aEEG) 3 recordings from 1310 comatose patients resuscitated after suffering from cardiac arrest, and admitted to the Intensive Care Unit of a large tertiary care hospital between February 2010 and April 2019.
The dataset was jointly annotated by two experts in cardiac arrest care and clinical EEG interpretation, using a simpliﬁed multinomial labeling convention to summarize EEG ﬁndings that occur commonly post cardiac arrest.
Speciﬁcally, each raw EEG record was broken into six disjoint 1-hour windows and classiﬁed as generalized suppression; burst suppression with epileptiform activity (including burst suppression with identical bursts); burst suppression without epileptiform activity; and, near-continuous or continuous background activity (Elmer et al., 2016; 2020).
We also developed a total of 17 LFs tailored to the prediction problem with help from one of the expert clinicians.
4. C OMPARISON O F M ETHODS
We compared WARM with the two existing active weak supervision approaches, namely Active
WeaSuL (Biegel et al., 2021) and the method proposed by Nashaat et al.
All the methods had access to the same set of initial LFs and were run for 30 active learning iterations, where they were allowed to query one ground truth label from the training set at each iteration.
We also benchmarked WARM against Snuba (Varma & R ´ e, 2018), a popular weak supervision approach which automatically synthesizes weak supervision sources from a small subset of labeled examples incorporates them with the data programming label model to probabilistically label data.
We ran Snuba with a random labeled subset of 10, 20 and 30 data points with decision tree stumps as heuristics.
To report test set performance, we trained downstream logistic regression models on the crisp versions of the probabilistic labels assigned by each method.
We also compared these methods against a logistic regression (LR) model trained using a vanilla active learning sampling strategy based on the distance from the decision boundary as in Biegel et al.
(2021), and a fully-supervised one trained using the entire labeled training set.
The fully supervised LR model serves as a proxy for the maximum achievable performance with the given features, train-test split and downstream classiﬁer.
Since existing online implementations of Snuba , Active WeaSuL and the method proposed by Nashaat et al.
(2018) were designed exclusively for binary classiﬁcation problems, we were only able to compare WARM with active learning and fully supervised baselines on the EEG dataset involving multi-class classiﬁcation.
For all our experiments, we used the logistic regression model implementation from Scikit-Learn (Pedregosa et al., 2011), trained for a maximum ofiterations and optimized using LBFGS.
WARM models were trained and built using PyTorch 1 . 8 .
Paszke et al.
along with Python 3 . 8 . 1 .
All our experiments were carried out on a Linux desktop with a typical machine having 8 Intel(R) Core(TM) i7-6700K CPUs and 16 GB of RAM.
[Figure]
4. E XPERIMENTAL D ESIGN
We aim to answer the following research questions through our experiments.
Does WARM improve the quality of the training dataset and how does this improvement translate to downstream classiﬁer performance?
We evaluate the quality of the training dataset by comparing the crisp versions of the probabilistic labels assigned by all the weakly supervised approaches with the ground truth training labels.
Can WARM combat the problem of knowledge shift and overcome noise in the initial LFs?
To this end, we carry out experiments on the synthetic and Heart Disease dataset.
To simulate knowledge shift in the synthetic dataset , we start with the three LFs tailored to the original (source) data distribution, and introduce two other (target) gaussian clusters with different centroids.
For the heart disease dataset, we automatically acquire LFs from the young population i.e. all patients with less than median age.
We then train and evaluate WARM on disjoint subsets of old patient population.
Since weakly supervised models rely heavily on the quality of its weak supervision sources, we conduct additional experiments on the Heart Disease dataset by adding varying levels of uniform random noise, scaled by the range of the features, to the LF decision parameters initially obtained using the random forest.
I.e. 100% noise added could mean that the decision stump threshold is off by 100%.
R ESULTS AND D ISCUSSION
Table 2 summarizes the results of our experiments.
The Train results represent the performance of the weak label models in comparison to ground truth training labels, and hence are a direct measure of label model quality.
The Test results on the other hand, are from LR models trained on the crisp equivalents of the probabilistic labels and evaluated on the held-out test set.
The latter set of results primarily reﬂect the quality of the downstream models trained using weak labels.
We refer the
[Figure]
[Table]
the Train set.
This improvement further translates into better downstream model performance on the Test set.
Fully Sup.
LR represents the performance of the fully-supervised logistic regression (LR) model trained with full access to ground truth labels.
interested reader to Appendix A.7 for supplementary results with additional metrics such as F1 and percentage change in accuracy.
Our primary ﬁndings from the experiments are summarized below.
WARM can improve the quality of training data.
We found that on all the datasets, the probabilistic labels assigned post active weak supervision via WARM are signiﬁcantly more accurate than the ones assigned by compared methods.
However, we did observe some variance in accuracy and its improvement across different datasets, primarily due to the nature of the data and the quality of initial weak supervision sources.
For instance on the breast cancer data, WARM improved the accuracy of the label model by over 7% across 30 active learning iterations.
On the other hand,
WARM could not improve the train generative performance on the heart disease data, but it also starts from a better higher accuracy.
We observed similar trends for downstream model performance, albeit with smaller accuracy improvements.
The smaller accuracy improvements can be attributed to the generalization capabilities of the downstream model over the labels it is trained on.
Nevertheless, our ﬁndings underscore the importance of high quality labeled training sets.
On the EEG dataset (Table 2 and Fig.
6–7), WARM achieved a 4 − 5% increase in both label model and downstream classiﬁer performance.
These results are signiﬁcant since even the initial set of LFs were developed by expert clinicians.
Moreover, we also found that traditional Active Learning strategy outperformed WARM on simpler classiﬁcation tasks such as breast cancer diagnosis.
However, for more complex prediction problems such as the predicting readmission on the Diabetes dataset and classifying clinical ﬁndings on the EEG dataset , active weak supervision via WARM signiﬁcantly outperformed end models trained using active learning.
Finally, experiment runtimes suggest that WARM is signiﬁcantly faster than existing active weak supervision techniques across datasets (Appendix A.5).
WARM can help combat knowledge shift and de-noise LFs.
Results from knowledge shift experiments on the heart disease and diabetes dataset in Fig.
2 and Table 4,5 reveal that WARM can help both weak labelers and the downstream classiﬁer adapt to different population characteristics.
We observed approximately a 1 − 3% and 4 − 6% increase in label and end model performance on the heart disease and synthetic datasets , respectively.
See Appendix A.6 for all LFs deﬁned on the heart disease dataset to detect the presence of heart disease in the young and old population.
Fig.
3(ii) and Tab. 6 summarizes the performance of WARM across 50 active learning iterations, when varying levels of random noise is artiﬁcially added to LFs deﬁned on the Wisconsin Diagnostic
Breast Cancer dataset .
Our results clearly show that noisy LFs hurt both label and end model performance.
However, WARM can iteratively de-noise LFs by actively collecting a few more labeled examples.
Even with 75% noise initially, WARM can reﬁne LFs such that the resulting label and end model perform on par with those trained using LFs with no noise.
Additionally, WARM consistently outperforms the other active weak supervision baselines 4 .
Changes to LFs brought about by WARM are interpretable.
If the LFs themselves are inherently interpretable, WARM supports their interpretability by enabling domain experts to track changes in their decision parameters.
Fig.
3(ii) for instance shows LFs before and after reﬁnement by WARM on the synthetic dataset .
We also draw the reader’s attention to the knowledge shift experiments on the heart disease dataset for more examples of interpretable changes to LF parameters.
Appendix A. lists all the LFs tailored to the young and old population.
For example, the rule If exercise induced angina <= 144.666(150.123) then ABSENCE else PRESENCE clearly says that while exercise induced angina higher than 144.67 was indicative of heart disease in the young population, older people tend to naturally have higher values resulting in a higher decision threshold.
Ablation experiments on WARM shows that it beneﬁts from both active learning and weak supervision .
We also perform ablation experiments to investigate if WARM is performing as expected.
In Appendix A.8, we see that WARM directly beneﬁts from Uncertainty Sampling from its active learning component, outperforming the random sampling in every case.
From Appendix A.10,A. it is generally able to generally outperform Active Learning on complex or large datasets due to its Weak Supervision component.
active learning results are not affected because we only add noise to LFs, which are only used by weak supervision methods
[Figure]
C ONCLUSION
With more medical data being collected, methods that address the lack of labeled data in clinical applications of machine learning are becoming increasingly vital.
If the vast amounts of unlabeled clinical data could be utilized to its full potential, the possibility of augmenting physicians with ML to increase positive patient outcomes is substantial.
In addition, a lot of of useful information can be provided to ML systems from expert guidance, whether it be via heuristic design or via handlabeling individual data points; making such capabilities efﬁcient and effective is greatly needed in practice.
WARM (Active Reﬁnement of Weakly Supervised Models) is a fast active learning addition to data programming that substantially and inexpensively improves the empirical accuracy of training labels and downstream classiﬁers.
Our experiments show that WARM has the capability to address the common problems of knowledge shift and rule denoising with minimal additional labeled data.
If ran on interpretable LFs, WARM is even able to reveal strategic, data-driven, interpretable updates to the LF thresholds and parameters.
Limitations of WARM include its reliance on differentiable and reasonably accurate initial LFs, which may occasionally be more expensive to acquire than pointillistic labels.
However, we have shown that most frequently used classes of LFs are intrinsically differentiable or can be made so.
WARM is already a step towards more efﬁcient and effective algorithms to tackle labeled-data-scarce ML problems in healthcare and beyond.
Additional experiments would beneﬁt precise assessment of properties and scoping of practical utility of WARM in a wider range of its possible applications.
Future work should also involve user studies to gauge attainable efﬁciency gains and utility of interpretable adjustments to the LFs, and whether they meaningfully boost perceived usability of the weak supervision process in practice.
Other future directions could include using meta-AL to learn query selection strategies based off of previous AL outcomes as in (Konyushkova et al., 2017; Bachman et al., 2017) or AL with partial feedback approaches as in (Hu et al., 2018).
Another challenge to address is mitigating overﬁtting to the active learning samples; we could perform early stopping, learning rate decay, gradient clipping, or cross validation to better generalize our models.
Furthermore, we would like to research the connection between our framework and automated (Varma & R´e, 2018) or active (Boecking et al., 2020) discovery of new LFs.
A A PPENDIX A. D ATASET D ETAILS
The Heart Disease dataset (Detrano et al., 1989) contains clinical and noninvasive test results from patients undergoing angiography at the Cleveland Clinic in Cleveland, Ohio, Hungarian Institute of
Cardiology in Budapest, Hungary, Veterans Administration Medical Center in Long Beach, California; and University Hospitals in Zurich and Basel, Switzerland.
For our experiments, we concatenate all the databases into one and predict whether or not an individual has heart disease.
The Diabetes dataset (Strack et al., 2014) comprises of clinical records from over million patients admitted to
US hospitals and integrated delivery networks between 1999 to 2008.
Given features representing patients such as their age, race, gender, time in hospital, diagnosis etc., the goal is to predict whether they are re-admitted for further treatment.
We also use two breast cancer datasets, namely Wisconsin
Diagnostic Breast Cancer dataset (Street et al., 1993) and Wisconsin Breast Cancer Database (Mangasarian & Wolberg, 1990).
For both the datasets, the prediction task is to distinguish between samples of malignant and benign breast cancer using different sets of features.
The diagnostic dataset comprises of features characterizing cell nuclei present in a digitized image of a ﬁne needle aspirate (FNA) of a breast mass.
The latter dataset instead contains 9 cytological characteristics of breast
FNA graded between 1–10 at the time of sample collection.
A. EEG D ATA C OLLECTION
EEG monitoring was initiated a median of 10 hours after initial collapse as standard routine of care on arrival to the intensive care unit (Elmer et al., 2016).
EEG was recorded using XLTech
Natus Neuroworks digital video/EEG systems (Natus Medical Inc) at 256Hz from 22 gold-plated cup electrodes placed in the international 10-20 system positions.
L¨ofhede et al.
(2008) has shown near-perfect spatial correlation of many clinically important EEG features in patients after cardiac arrest, allowing the 22-sample signal to be down-sampled without signiﬁcant information loss.
Additionally, after cardiac arrest, many prognostic clinical ﬁndings can be identiﬁed accurately from lower-resolution quantitative summaries (e.g. 1Hz aEEG, Fig. 4)
(Oh et al., 2015; Glass et al., 2013).
Thus, to optimize computational efﬁciency, we aimed to develop similar heuristics as labeling functions to classify raw EEG signals from contemporaneous aEEG summaries
[Table] [Table]
For our experiments, we used 6 -hour aEEG time series starting at the initiation of EEG monitoring from 1310 patients with mean age 57 years of whom 503 ( 38% ) were female.
Approximately 30% survived to discharge from the hospital.
A large subset of our data (approximately 70% ) was initially jointly annotated by two experts in cardiac arrest care and clinical EEG interpretation at
UPMC.
They manually annotated EEG ﬁndings using an adaptation of the 2016 American Clinical
Neurophysiology Society guidelines (Hirsch et al., 2013).
Based on prior research and our expert clinicians, we developed a simpliﬁed multinomial labeling convention to summarize prognostic EEG ﬁndings that occur commonly after cardiac arrest.
Specifically, we classiﬁed the raw EEG waveforms as: generalized suppression; burst suppression with epileptiform activity (including burst suppression with identical bursts); burst suppression without epileptiform activity; and, near-continuous or continuous background activity (Elmer et al., 2020).
Because continuous background activity with epileptiform activity occurred infrequently in this data set, we did not further subdivide categories of patients with continuous background activity.
A
[Table]
K NOWLEDGE S HIFT E XPERIMENTS
[Table] [Table]
A
[Table]
A DDING A RTIFICIAL N OISE TO LF S
[Table]
to ﬁnal accuracy.
Noisy LFs hurt both label and end model performance.
However, WARM can iteratively de-noise LFs by actively collecting a few more labeled examples.
Even with 75% noise initially, WARM can reﬁne LFs such that the resulting label and end model perform on par with those trained using LFs with no noise.
WARM also consistently outperforms other baselines.
[Figure]
A. E XAMPLE R ULES
On the heart disease dataset , we carried out our knowledge shift experiments, where the initial labeling functions were based on the young population, but the target population was that of old patients.
Below is a list of initial rules tailored to the young patients.
The decision parameters in parenthesis reﬂect the modiﬁed labeling function parameters, tailored to older patients.
If Chest pain type <= 0.898(0.978) then ABSENCE else PRESENCE
If Serum cholesterol in mg/dl <= 124.057(119.793) then PRESENCE else ABSENCE
If Maximum heart rate achieved <= 0.244(0.079) then PRESENCE else ABSENCE
If Exercise induced angina <= 144.666(150.123) then ABSENCE else PRESENCE
If thal <= 0.646(0.796) then ABSENCE else PRESENCE
If Chest pain type <= 0.898(0.978) then ABSENCE else PRESENCE
If ST depression induced by exercise relative to rest <= 0.273(0.383) then ABSENCE else PRESENCE
If Exercise induced angina <= 144.666(150.123) then ABSENCE else PRESENCE
If thal <= 0.646(0.796) then ABSENCE else PRESENCE
If Maximum heart rate achieved <= 0.533(0.367) then PRESENCE else ABSENCE
[Figure]
THER
[Figure]
[Table] [Figure]
[Table]
[Figure]
[Table]
[Figure] [Figure]
[Table] [Table]
[Table] [Table]
[Table]
A. WARM R ANDOM S AMPLING VS U NCERTAINTY S AMPLING
[Table]
[Table]
A. WARM A BLATION E XPERIMENTS ON D ATASET S IZE AND D OWNSTREAM M ODELS For
the rest of our experiments we created synthetic datasets using sklearn ’s make classification 5 method.
The datasets comprise of clusters of points normally distributed (with standard deviation 1) about vertices of a 25-dimensional hypercube with sides of length 2 , with an equal number of clusters ( = 5 ) to each class.
We used 25 decision tree stumps as LFs.
In order to automatically create the simple LFs, we followed the same procedure outlined in Section 4.1, with the only difference being that the Random Forest classiﬁer was trained using 250 data points sampled from the training dataset.
This was done to ensure that WARM received the same amount of weak supervision regardless of the size of the dataset
[Table]
and the differences in performance were solely due to the size of the dataset and not the quality of weak supervision
[Table]
A. WARM E XPERIMENTS ON C OMPLEXITY OF D ATASET
The following results are reported on a datasets with 10000 samples and varying numbers of clusters per class, created using sklearn ’s make classification method.
The number of clusters per class controls the complexity of the problem, and more clusters per class is a harder classiﬁcation problem than fewer clusters per class.
The weak supervision sources were created in exactly the same way as mentioned in the previous section.
[Table]
Active learning
[Table]
Active learning
[Table]
Active learning
[Table]