Original: I NTERACTIVE W EAK S UPERVISION : L EARNING U SEFUL H EURISTICS FOR D ATA L ABELING
Revised: I NTERACTIVE W EAK S UPERVISION : L EARNING U SEFUL H EURISTICS FOR D ATA L ABELING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Obtaining large annotated datasets is critical for training successful machine learning models and it is often a bottleneck in practice.
Revised: Obtaining large annotated datasets is critical for training successful machine learning models and it is often a bottleneck in practice.

Original: Weak supervision offers a promising alternative for producing labeled datasets without ground truth annotations by generating probabilistic labels using multiple noisy heuristics.
Revised: Weak supervision offers a promising alternative for producing labeled datasets without ground truth annotations by generating probabilistic labels using multiple noisy heuristics.

Original: This process can scale to large datasets and has demonstrated state of the art performance in diverse domains such as healthcare and e-commerce.
Revised: This process can scale to large datasets and has demonstrated state of the art performance in diverse domains such as healthcare and e-commerce.

Original: One practical issue with learning from user-generated heuristics is that their creation requires creativity, foresight, and expertise from those who hand-craft them, a process which can be tedious and subjective.
Revised: One practical issue with learning from user-generated heuristics is that their creation requires creativity, foresight, and expertise from those who hand-craft them, a process which can be tedious and subjective.

Original: We develop the ﬁrst framework for interactive weak supervision in which a method proposes heuristics and learns from user feedback given on each proposed heuristic.
Revised: We develop the ﬁrst framework for interactive weak supervision in which a method proposes heuristics and learns from user feedback given on each proposed heuristic.

Original: Our experiments demonstrate that only a small number of feedback iterations are needed to train models that achieve highly competitive test set performance without access to ground truth training labels.
Revised: Our experiments demonstrate that only a small number of feedback iterations are needed to train models that achieve highly competitive test set performance without access to ground truth training labels.

Original: We conduct a real user study, which shows that users are able to effectively provide feedback on heuristics and that test set results track the performance of simulated oracles.
Revised: We conduct a real user study, which shows that users are able to effectively provide feedback on heuristics and that test set results track the performance of simulated oracles.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: The performance of supervised machine learning (ML) hinges on the availability of labeled data in sufﬁcient quantity and quality.
Revised: The performance of supervised machine learning (ML) hinges on the availability of labeled data in sufﬁcient quantity and quality.

Original: However, labeled data for applications of ML can be scarce, and the common process of obtaining labels by having annotators inspect individual samples is often expensive and time consuming.
Revised: However, labeled data for applications of ML can be scarce, and the common process of obtaining labels by having annotators inspect individual samples is often expensive and time consuming.

Original: Additionally, this cost is frequently exacerbated by factors such as privacy concerns, required expert knowledge, and shifting problem deﬁnitions.
Revised: Additionally, this cost is frequently exacerbated by factors such as privacy concerns, required expert knowledge, and shifting problem deﬁnitions.

Original: Weak supervision provides a promising alternative, reducing the need for humans to hand label large datasets to train machine learning models (Riedel et al., 2010; Hoffmann et al., 2011; Ratner et al., 2016).
Revised: Weak supervision provides a promising alternative, reducing the need for humans to hand label large datasets to train machine learning models (Riedel et al., 2010; Hoffmann et al., 2011; Ratner et al., 2016).

Original: A recent approach called data programming (Ratner et al., 2016)
Revised: A recent approach called data programming (Ratner et al., 2016)

Original: combines multiple weak supervision sources by using an unsupervised label model to estimate the latent true class label, an idea that has close connections to modeling workers in crowd-sourcing (Dawid & Skene, 1979;
Revised: combines multiple weak supervision sources by using an unsupervised label model to estimate the latent true class label, an idea that has close connections to modeling workers in crowd-sourcing (Dawid & Skene, 1979;

Original: Karger et al., 2011; Dalvi et al., 2013; Zhang et al., 2014).
Revised: Karger et al., 2011; Dalvi et al., 2013; Zhang et al., 2014).

Original: The approach enables subject matter experts to specify labeling functions (LFs) —functions that encode domain knowledge and noisily annotate subsets of data, such as user-speciﬁed heuristics or external knowledge bases—instead of needing to inspect and label individual samples.
Revised: The approach enables subject matter experts to specify labeling functions (LFs) —functions that encode domain knowledge and noisily annotate subsets of data, such as user-speciﬁed heuristics or external knowledge bases—instead of needing to inspect and label individual samples.

Original: These weak supervision approaches have been used on a wide variety of data types such as MRI sequence and unstructured text, and in various domains such as healthcare and e-commerce (Fries et al., 2019; Halpern et al., 2014; Bach et al., 2019; R´e et al., 2020).
Revised: These weak supervision approaches have been used on a wide variety of data types such as MRI sequence and unstructured text, and in various domains such as healthcare and e-commerce (Fries et al., 2019; Halpern et al., 2014; Bach et al., 2019; R´e et al., 2020).

Original: Not only does the use of multiple sources of weak supervision provide a scalable framework for creating large labeled datasets, but it can also be viewed as a vehicle to incorporate high level, conceptual feedback into the data labeling process.
Revised: Not only does the use of multiple sources of weak supervision provide a scalable framework for creating large labeled datasets, but it can also be viewed as a vehicle to incorporate high level, conceptual feedback into the data labeling process.

Original: In data programming, each LF is an imperfect but reasonably accurate heuristic, such as a pre-trained classiﬁer or keyword lookup.
Revised: In data programming, each LF is an imperfect but reasonably accurate heuristic, such as a pre-trained classiﬁer or keyword lookup.

Original: For example, for the popular 20 newsgroups dataset, an LF to identify the class ‘ sci.space ’ may look for the token ‘ launch ’ in documents and would be right about 70% of the time.
Revised: For example, for the popular 20 newsgroups dataset, an LF to identify the class ‘ sci.space ’ may look for the token ‘ launch ’ in documents and would be right about 70% of the time.

Original: While data programming can be very effective when done right, experts may spend a signiﬁcant amount of time designing the weak supervision sources (Varma & R´e, 2018) and must often inspect samples at random to generate ideas (Cohen-Wang et al., 2019).
Revised: While data programming can be very effective when done right, experts may spend a signiﬁcant amount of time designing the weak supervision sources (Varma & R´e, 2018) and must often inspect samples at random to generate ideas (Cohen-Wang et al., 2019).

Original: In our 20 newsgroups example, we may randomly see a document mentioning ‘ Salman Rushdie ’ and realize that the name of a famous atheist could be a good heuristic to identify posts in ‘ alt.atheism ’.
Revised: In our 20 newsgroups example, we may randomly see a document mentioning ‘ Salman Rushdie ’ and realize that the name of a famous atheist could be a good heuristic to identify posts in ‘ alt.atheism ’.

Original: While such a heuristic seems obvious after the fact, we have to chance upon the right documents to generate these ideas.
Revised: While such a heuristic seems obvious after the fact, we have to chance upon the right documents to generate these ideas.

Original: In
Revised: In

Original: IWS chooses a ﬁnal set of LFs, which are used to estimate labels for X and train an end classiﬁer ... ...
Revised: IWS chooses a ﬁnal set of LFs, which are used to estimate labels for X and train an end classiﬁer ... ...

Original: Interactive Weak Supervision (IWS) proposes LFs from to experts and learns from feedback on these LFs
Revised: Interactive Weak Supervision (IWS) proposes LFs from to experts and learns from feedback on these LFs

Original: Yes / No
Revised: Yes / No

Original: Update IWS model
Revised: Update IWS model

Original: Query: “Is better than random?”
Revised: Query: “Is better than random?”

Original: An LF family generates a large set .
Revised: An LF family generates a large set .

Original: of candidate heuristics (LFs) for an unlabeled dataset X
Revised: of candidate heuristics (LFs) for an unlabeled dataset X

Original: [Figure]
Revised: [Figure]

Original: z
Revised: z

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: ,L):
Revised: ,L):

Original: [Figure]
Revised: [Figure]

Original: (L
Revised: (L

Original: [Figure]
Revised: [Figure]

Original: practice, coming up with effective LFs becomes difﬁcult after the ﬁrst few.
Revised: practice, coming up with effective LFs becomes difﬁcult after the ﬁrst few.

Original: Substantial foresight and creativity are required to create a function that applies to a non-negligible subset of given data, is novel, and adds predictive value.
Revised: Substantial foresight and creativity are required to create a function that applies to a non-negligible subset of given data, is novel, and adds predictive value.

Original: We propose a new approach for training supervised ML models by interactively discovering weak supervision sources.
Revised: We propose a new approach for training supervised ML models by interactively discovering weak supervision sources.

Original: Our method supports domain experts in fast discovery of good LFs.
Revised: Our method supports domain experts in fast discovery of good LFs.

Original: The method queries users in an active fashion for feedback about candidate LFs, from which a model learns to distinguish LFs likely to have good accuracy.
Revised: The method queries users in an active fashion for feedback about candidate LFs, from which a model learns to distinguish LFs likely to have good accuracy.

Original: Upon completion, our method produces a ﬁnal set of
Revised: Upon completion, our method produces a ﬁnal set of

Original: LFs.
Revised: LFs.

Original: We use this set to create an estimate of the latent class label via an unsupervised label model and train the ﬁnal supervised ML model using a noise aware loss function on the estimated labels as in Ratner et al.
Revised: We use this set to create an estimate of the latent class label via an unsupervised label model and train the ﬁnal supervised ML model using a noise aware loss function on the estimated labels as in Ratner et al.

Original: The approach relies on the observation that many applications allow for heuristics of varying quality to be generated at scale, similar to Varma & R ´ e (2018), and that experts can provide good judgment by identifying some LFs that have reasonable accuracy.
Revised: The approach relies on the observation that many applications allow for heuristics of varying quality to be generated at scale, similar to Varma & R ´ e (2018), and that experts can provide good judgment by identifying some LFs that have reasonable accuracy.

Original: The full pipeline of the proposed approach, termed Interactive Weak Supervision (IWS) , is illustrated in Fig.
Revised: The full pipeline of the proposed approach, termed Interactive Weak Supervision (IWS) , is illustrated in Fig.

Original: Our contributions are:
Revised: Our contributions are:

Original: We propose, to the best of our knowledge, the ﬁrst interactive method for weak supervision in which queries to be annotated are not data points but labeling functions.
Revised: We propose, to the best of our knowledge, the ﬁrst interactive method for weak supervision in which queries to be annotated are not data points but labeling functions.

Original: This approach automates the design of useful data labeling heuristics.
Revised: This approach automates the design of useful data labeling heuristics.

Original: We conduct experiments with real users on two text classiﬁcation datasets.
Revised: We conduct experiments with real users on two text classiﬁcation datasets.

Original: Results support our modeling assumptions, and show that users can provide accurate feedback on automatically generated LFs.
Revised: Results support our modeling assumptions, and show that users can provide accurate feedback on automatically generated LFs.

Original: Our method shows superior performance compared to standard active learning, meaning we can achieve better test set performance with a smaller number of queries to users.
Revised: Our method shows superior performance compared to standard active learning, meaning we can achieve better test set performance with a smaller number of queries to users.

Original: After 200 IWS iterations annotating LFs, we achieve a mean test set AUC that requires at least three times as many active learning iterations annotating samples.
Revised: After 200 IWS iterations annotating LFs, we achieve a mean test set AUC that requires at least three times as many active learning iterations annotating samples.

Original: In addition, the average user response time for IWS queries is shorter than for data-point-wise active learning queries.
Revised: In addition, the average user response time for IWS queries is shorter than for data-point-wise active learning queries.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Active strategies for weak supervision sources have largely focused on combinations of data programming (Ratner et al., 2016) with traditional active learning where samples are presented to users.
Revised: Active strategies for weak supervision sources have largely focused on combinations of data programming (Ratner et al., 2016) with traditional active learning where samples are presented to users.

Original: In Nashaat et al.
Revised: In Nashaat et al.

Original: (2018), a pool of samples is created on which LFs disagree, and active learning strategies are then applied to obtain labels for some of these data points.
Revised: (2018), a pool of samples is created on which LFs disagree, and active learning strategies are then applied to obtain labels for some of these data points.

Original: In Cohen-Wang et al.
Revised: In Cohen-Wang et al.

Original: (2019), speciﬁc samples are selected and presented to users in order to inspire the creation of LFs.
Revised: (2019), speciﬁc samples are selected and presented to users in order to inspire the creation of LFs.

Original: In Hancock et al.
Revised: In Hancock et al.

Original: (2018), natural language explanations provided during text labeling are used to generate heuristics.
Revised: (2018), natural language explanations provided during text labeling are used to generate heuristics.

Original: The proposed system uses a semantic parser to convert explanations into logical forms, which represent labeling functions.
Revised: The proposed system uses a semantic parser to convert explanations into logical forms, which represent labeling functions.

Revised: Beyond weak supervision, Srivastava et al.

Original:  Prior work has emphasized that LFs deﬁned by experts frequently have a recurring structure in which elements are swapped to change the higher level concept a function corresponds to (Varma & R´e, 2018; Varma et al., 2017; Bach et al., 2019).
Revised: (2019) introduce a reinforcement learning framework for learning via natural language in a conversational setting. Prior work has emphasized that LFs deﬁned by experts frequently have a recurring structure in which elements are swapped to change the higher level concept a function corresponds to (Varma & R´e, 2018; Varma et al., 2017; Bach et al., 2019).

Original: As an example, in tasks involving text documents,
Revised: As an example, in tasks involving text documents,

Original: LFs often follow a repetitive structure in which key terms or phrases and syntactical relationships change, e.g. mentions of speciﬁc words (Varma & R´e, 2018; Cohen-Wang et al., 2019; Varma et al., 2019).
Revised: LFs often follow a repetitive structure in which key terms or phrases and syntactical relationships change, e.g. mentions of speciﬁc words (Varma & R´e, 2018; Cohen-Wang et al., 2019; Varma et al., 2019).

Original: Prior work relies on this observation to create heuristic generators (Varma & R´e, 2018), LF templates (Bach et al., 2019), and domain-speciﬁc primitives (Varma et al., 2017).
Revised: Prior work relies on this observation to create heuristic generators (Varma & R´e, 2018), LF templates (Bach et al., 2019), and domain-speciﬁc primitives (Varma et al., 2017).

Original: In particular, in a semi-supervised data programming setting, Varma & R´e (2018) propose a system for automatic generation of labeling functions without user interaction, by using a small set of labeled data.
Revised: In particular, in a semi-supervised data programming setting, Varma & R ´ e (2018) propose a system for automatic generation of labeling functions without user interaction, by using a small set of labeled data.

Original: The authors motivate their system stating that users frequently perform repetitive steps such as guessing optimal numerical thresholds and developing informative text patterns.
Revised: The authors motivate their system stating that users frequently perform repetitive steps such as guessing optimal numerical thresholds and developing informative text patterns.

Revised: In the growing body of literature on learning from multiple weak supervision sources further related work has studied weak supervision for neural networks in information retrieval (Dehghani et al., 2017;

Original: Additional related work has focused on the modeling of dependencies among heuristics in data programming (Bach et al., 2017; Varma et al., 2019),on the multi-task  setting (Ratner et al., 2019),on handling of multi-resolution sources (Sala et al., 2019),on the use of noisy pairwise labeling functions (Boecking & Dubrawski, 2019),on addressing latent subsets in the data (Varma et al., 2016), andon fast model iteration via the use of pre-trained embeddings (Chen et al., 2020). 
Revised: Zamani et al., 2018; Zamani & Croft, 2018), the modeling of dependencies among heuristics in data programming (Bach et al., 2017; Varma et al., 2019), the multi-task data programming setting (Ratner et al., 2019), handling of multi-resolution sources (Sala et al., 2019), the use of noisy pairwise labeling functions (Boecking & Dubrawski, 2019), addressing latent subsets in the data (Varma et al., 2016), and fast model iteration via the use of pre-trained embeddings (Chen et al., 2020). Finally, Dehghani et al. (2018) offer insight into how the quality of weak labels can be reﬁned on a per-sample basis.

Original:  M ETHODS
Revised:  M ETHODS

Original: We propose an interactive weak supervision (IWS) approach to assist experts in ﬁnding good labeling functions (LFs) for training a classiﬁer on datasets without ground truth labels.
Revised: We propose an interactive weak supervision (IWS) approach to assist experts in ﬁnding good labeling functions (LFs) for training a classiﬁer on datasets without ground truth labels.

Original: We will ﬁrst describe the general problem setting of learning to classify without ground truth samples by modeling multiple weak supervision sources, as well as the concept of LF families.
Revised: We will ﬁrst describe the general problem setting of learning to classify without ground truth samples by modeling multiple weak supervision sources, as well as the concept of LF families.

Original: We then dive into the details of the proposed IWS approach.
Revised: We then dive into the details of the proposed IWS approach.

Original: For brevity, we limit the scope of the end classiﬁer to binary classiﬁcation, but the presented background and ideas extend to the multi-class settings.
Revised: For brevity, we limit the scope of the end classiﬁer to binary classiﬁcation, but the presented background and ideas extend to the multi-class settings.

Original: 3. P RELIMINARIES
Revised: 3. P RELIMINARIES

Original: Learning with Multiple Weak Supervision Sources Assume each data point x ∈ X has a latent class label y ∗ ∈ Y
Revised: Learning with Multiple Weak Supervision Sources Assume each data point x ∈ X has a latent class label y ∗ ∈ Y

Original: = {− 1 , 1 } .
Revised: = {− 1 , 1 } .

Original: Given n unlabeled, i.i.d.
Revised: Given n unlabeled, i.i.d.

Original: datapoints
Revised: datapoints

Original: X = { x i } ni =1 , our goal is to train an end classiﬁer f : X → Y such that f ( x ) =
Revised: X = { x i } ni =1 , our goal is to train an end classiﬁer f : X → Y such that f ( x ) =

Original: y ∗ .
Revised: y ∗ .

Original: In data programming (Ratner et al., 2016; 2020), a user provides m LFs { λ j } mj =1 , where λ j : X → Y ∪ { 0 } noisily label the data with λ j ( x ) ∈ {− 1 , 1 } or abstain with λ j ( x ) = 0 .
Revised: In data programming (Ratner et al., 2016; 2020), a user provides m LFs { λ j } mj =1 , where λ j : X → Y ∪ { 0 } noisily label the data with λ j ( x ) ∈ {− 1 , 1 } or abstain with λ j ( x ) = 0 .

Original: The corresponding LF output matrix is Λ ∈ {− 1 , 0 , 1 } n × m , where Λ i,j = λ j ( x i ) .
Revised: The corresponding LF output matrix is Λ ∈ {− 1 , 0 , 1 } n × m , where Λ i,j = λ j ( x i ) .

Original: In this paper, we assume that each LF λ j has the same accuracy on each class, α j = P ( λ j ( x ) =
Revised: In this paper, we assume that each LF λ j has the same accuracy on each class, α j = P ( λ j ( x ) =

Original: y ∗ | λ j ( x ) (cid:54) = 0) , where accuracy is deﬁned on items where j does not abstain.
Revised: y ∗ | λ j ( x ) (cid:54) = 0) , where accuracy is deﬁned on items where j does not abstain.

Original: Further, we denote by l j = P ( λ j ( x ) (cid:54) = 0)
Revised: Further, we denote by l j = P ( λ j ( x ) (cid:54) = 0)

Original: the LF propensity (sometimes called LF coverage), i.e. the frequency at which LF j does not abstain.
Revised: the LF propensity (sometimes called LF coverage), i.e. the frequency at which LF j does not abstain.

Original: In data programming, an unsupervised label model p θ ( Y, Λ) produces probabilistic estimates of the latent class labels Y ∗
Revised: In data programming, an unsupervised label model p θ ( Y, Λ) produces probabilistic estimates of the latent class labels Y ∗

Original: = { y ∗ i } ni =1 using observed Λ by modeling the LF accuracies, propensities, and possibly their dependencies.
Revised: = { y ∗ i } ni =1 using observed Λ by modeling the LF accuracies, propensities, and possibly their dependencies.

Original: A number of label model approaches exist in the crowd-sourcing (Dawid & Skene, 1979; Zhang et al., 2014) and the weak supervision literature (Ratner et al., 2020).
Revised: A number of label model approaches exist in the crowd-sourcing (Dawid & Skene, 1979; Zhang et al., 2014) and the weak supervision literature (Ratner et al., 2020).

Original: In this paper, we use a factor graph as proposed in Ratner et al.
Revised: In this paper, we use a factor graph as proposed in Ratner et al.

Original: (2016; 2020) to obtain probabilistic labels by modeling the LF accuracies via factor φ Labi,j (Λ , Y ) (cid:44) 1 { Λ ij (cid:54) = 0 } and labeling propensity by factor φ Acci,j (Λ , Y ) (cid:44) 1 { Λ ij = y i } , and for simplicity assume LFs are independent conditional on Y .
Revised: (2016; 2020) to obtain probabilistic labels by modeling the LF accuracies via factor φ Labi,j (Λ , Y ) (cid:44) 1 { Λ ij (cid:54) = 0 } and labeling propensity by factor φ Acci,j (Λ , Y ) (cid:44) 1 { Λ ij = y i } , and for simplicity assume LFs are independent conditional on Y .

Original: The label model is deﬁned as
Revised: The label model is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: where Z θ is a normalizing constant and φ i (Λ i , y i ) deﬁned to be the concatenation of the factors for all LFs j = 1 , . . .
Revised: where Z θ is a normalizing constant and φ i (Λ i , y i ) deﬁned to be the concatenation of the factors for all LFs j = 1 , . . .

Original: , m for sample i .
Revised: , m for sample i .

Original: We learn θ by minimizing the negative log marginal likelihood given the observed Λ .
Revised: We learn θ by minimizing the negative log marginal likelihood given the observed Λ .

Original: Finally, following Ratner et al.
Revised: Finally, following Ratner et al.

Original: the end classiﬁer f is trained using probabilistic labels p θ ( Y | Λ) .
Revised: the end classiﬁer f is trained using probabilistic labels p θ ( Y | Λ) .

Original: Labeling Function Families We deﬁne LF families as sets of expert-interpretable weak supervision sources described by functions z φ : X (cid:55)→ {− 1 , 0 , 1 } , for parameters φ ∈ Φ .
Revised: Labeling Function Families We deﬁne LF families as sets of expert-interpretable LFs described by functions z φ : X (cid:55)→ {− 1 , 0 , 1 } , for parameters φ ∈ Φ .

Original: An example are shallow decision trees z φ parameterized by variables or splitting rules φ (Varma & R´e, 2018), or a function z φ deﬁning a regular expression for two words where φ parameterizes the word choices from a vocabulary Φ .
Revised: An example are shallow decision trees z φ parameterized by variables or splitting rules φ (Varma & R´e, 2018), or a function z φ deﬁning a regular expression for two words where φ parameterizes the word choices from a vocabulary Φ .

Original: Given such an LF family, we can generate a large set of p candidate heuristics L = { λ j ( x ) =
Revised: Given such an LF family, we can generate a large set of p candidate heuristics L = { λ j ( x ) =

Original: z φ j ( x ) } p j =1 , where φ j ∈ Φ , e.g. by sampling from Φ and pruning low coverage candidates.
Revised: z φ j ( x ) } p j =1 , where φ j ∈ Φ , e.g. by sampling from Φ and pruning low coverage candidates.

Revised: These families will frequently arise naturally in the form of LFs with repetitive structure that experts usually write from scratch, where template variables—such as words or time series landmarks—can be sampled from the unlabeled data to create candidates.

Original: For text, we can ﬁnd n-grams within a document frequency range to generate key term lookups, ﬁll placeholders in regular expressions, or generate shallow decision trees (Ratner et al., 2016; Varma & R´e, 2018; Varma et al., 2019).
Revised: For text, we can ﬁnd n-grams within a document frequency range to generate key term lookups, ﬁll placeholders in regular expressions, or generate shallow decision trees (Ratner et al., 2016; Varma & R´e, 2018; Varma et al., 2019).

Original: For time series, we can create a large set of LFs based on motifs (Lonardi & Patel, 2002) or graphs of temporal constraints (Guillame-Bert & 
Revised: For time series, we can create a large set of LFs based on motifs (Lonardi & Patel, 2002) or graphs of temporal constraints (Guillame-Bert & Dubrawski, 2017).

Original: Dubrawski, 2017).
Revised: For images, we can create a library pre-trained object detectors as in (Chen et al., 2019). Varma & R´e (2018) show that some image tasks also lend themselves to weak supervision sources generated from geometric properties of the images. An LF family has to be chosen with domain expert input. Compared to standard data programming, the burden of creating LFs from scratch is shifted to choosing an appropriate LF family and then judging recommended candidates.

Original: For images, we can create a library of classiﬁers pre-trained for various speciﬁc tasks (e.g. object detectors as used in (Chen et al., 2019)).
Revised: However, domain experts often have foresight enabling the choice of an LF family such that a sufﬁciently sized subset of LFs is predictive of the latent class label. Such

Original:  3. I NTERACTIVE W EAK S UPERVISION
Revised: LF families may not exist for all data types and classiﬁcation tasks. But when they exist they offer the opportunity to quickly build large, labeled datasets. Further, once created, it is reasonable to expect that the same LF generation procedure can be reused for similar classiﬁcation tasks without additional effort (e.g. we use a single procedure for all text LFs in our experiments). 3. I NTERACTIVE W EAK S UPERVISION

Original: Instead of having users provide m good weak supervision sources up front, we want to assist users in discovering them.
Revised: Instead of having users provide m good weak supervision sources up front, we want to assist users in discovering them.

Original: Successful applications of data programming have established that human experts are able to construct accurate LFs from scratch.
Revised: Successful applications of data programming have established that human experts are able to construct accurate LFs from scratch.

Original: Our work leverages the assumption that human experts can also judge these properties when presented with pre-generated LFs of the same form.
Revised: Our work leverages the assumption that human experts can also judge these properties when presented with pre-generated LFs of the same form.

Original: Suppose again that we have an unlabeled dataset X
Revised: Suppose again that we have an unlabeled dataset X

Original: = { x i } ni
Revised: = { x i } ni

Original: =1 , and that our goal is to train an end classiﬁer f without access to labels Y ∗ =
Revised: =1 , and that our goal is to train an end classiﬁer f without access to labels Y ∗ =

Original: { y ∗ i } ni =1 .
Revised: { y ∗ i } ni =1 .

Original: Assume also that we deﬁned a large pool of p candidate LFs L = { λ j ( x ) } pj =1 from an LF family (following Sec. 3.1), of varying accuracy and coverage.
Revised: Assume also that we deﬁned a large pool of p candidate LFs L = { λ j ( x ) } pj =1 from an LF family (following Sec. 3.1), of varying accuracy and coverage.

Original: In IWS, our goal is to identify an optimal subset of LFs L ∗ ⊂
Revised: In IWS, our goal is to identify an optimal subset of LFs L ∗ ⊂

Original: L to pass to the label model in Eq.
Revised: L to pass to the label model in Eq.

Original: Below we will quantify how L ∗ depends on certain properties of LFs.
Revised: Below we will quantify how L ∗ depends on certain properties of LFs.

Original: While we can observe some of these properties—such as coverage, agreement, and conﬂicts—an important property that we cannot observe is the accuracy of each LF.
Revised: While we can observe some of these properties—such as coverage, agreement, and conﬂicts—an important property that we cannot observe is the accuracy of each LF.

Original: Our goal will thus be to infer quantities related to the latent accuracies α j ∈ [0 , 1] of LFs λ j ∈ L , given a small amount expert feedback.
Revised: Our goal will thus be to infer quantities related to the latent accuracies α j ∈ [0 , 1] of LFs λ j ∈ L , given a small amount expert feedback.

Original: To do this, we deﬁne an expert-feedback model , which can be used to infer LF accuracies given a dataset of user feedback.
Revised: To do this, we deﬁne an expert-feedback model , which can be used to infer LF accuracies given a dataset of user feedback.

Original: To efﬁciently train this model, our
Revised: To efﬁciently train this model, our

Original: IWS procedure sequentially chooses an LF λ j ∈ L and shows a description of λ j to an expert, who provides binary feedback about λ j .
Revised: IWS procedure sequentially chooses an LF λ j ∈ L and shows a description of λ j to an expert, who provides binary feedback about λ j .

Original: We follow ideas from active learning for sequential decision making under uncertainty, in which a probabilistic model guides data collection to efﬁciently infer quantities of interest within T iterations.
Revised: We follow ideas from active learning for sequential decision making under uncertainty, in which a probabilistic model guides data collection to efﬁciently infer quantities of interest within T iterations.

Original: After a sequence of feedback iterations, we use the expertfeedback model to provide an estimate ˆ L ⊂ L of the optimal subset L ∗ .
Revised: After a sequence of feedback iterations, we use the expertfeedback model to provide an estimate ˆ L ⊂ L of the optimal subset L ∗ .

Original: The label model then uses ˆ L to produce a probabilistic estimate of Y ∗ , which is used to train the end classiﬁer f .
Revised: The label model then uses ˆ L to produce a probabilistic estimate of Y ∗ , which is used to train the end classiﬁer f .

Original: The full IWS procedure is shown in the ﬂow chart in Fig.
Revised: The full IWS procedure is illustrated in Fig.

Original: 1 which we describe in detail below.
Revised: 1 and described in detail below.

Original: Expert-Feedback Model
Revised: Expert-Feedback Model

Original: We ﬁrst deﬁne a generative model of human expert feedback about LFs, given the latent LF accuracies.
Revised: We ﬁrst deﬁne a generative model of human expert feedback about LFs, given the latent LF accuracies.

Original: This model will form the basis for an online procedure that selects a sequence of LFs to show to human experts.
Revised: This model will form the basis for an online procedure that selects a sequence of LFs to show to human experts.

Original: We task experts to classify LFs as either useful or not useful u j ∈ { 0 , 1 } , corresponding to their belief
Revised: We task experts to classify LFs as either useful or not useful u j ∈ { 0 , 1 } , corresponding to their belief

Original: that LF λ
Revised: that LF λ

Original: j is predictive of Y ∗ at better than random accuracy for the samples where λ j does not abstain .
Revised: j is predictive of Y ∗ at better than random accuracy for the samples where λ j does not abstain .

Original: Note that prior data programming work (Ratner et al., 2016; 2019; Dunnmon et al., 2020; Saab et al., 2020) assumes and demonstrates that experts are able to use their domain knowledge to make this judgment when creating LFs from scratch.
Revised: Note that prior data programming work (Ratner et al., 2016; 2019; Dunnmon et al., 2020; Saab et al., 2020) assumes and demonstrates that experts are able to use their domain knowledge to make this judgment when creating LFs from scratch.

Original: We model the generative process for this feedback and the latent LF accuracies as, for j = 1 , . . .
Revised: We model the generative process for this feedback and the latent LF accuracies as, for j = 1 , . . .

Original: , t :
Revised: , t :

Original: [Equation]
Revised: [Equation]

Original: where v j can be viewed as the average probability that a human will label a given LF λ
Revised: where v j can be viewed as the average probability that a human will label a given LF λ

Original: j as u j = 1 , and h ω ( λ j ) is a parameterized function (such as a neural network), mapping each LF λ
Revised: j as u j = 1 , and h ω ( λ j ) is a parameterized function (such as a neural network), mapping each LF λ

Original: j to v j .
Revised: j to v j .

Original: Finally, to model the connection between accuracy α j and v j , we assume that v j = g ( α j ) , where g : [0 , 1] → [0 , 1] is a monotonic increasing function mapping unknown LF accuracy α j to v j .
Revised: Finally, to model the connection between accuracy α j and v j , we assume that v j = g ( α j ) , where g : [0 , 1] → [0 , 1] is a monotonic increasing function mapping unknown LF accuracy α j to v j .

Original: After t queries of human feedback, we have produced a query dataset Q t = { ( λ j , u j ) } tj =1 .
Revised: After t queries of human feedback, we have produced a query dataset Q t = { ( λ j , u j ) } tj =1 .

Original: Given
Revised: Given

Original: Q t , we will infer unknown quantities in the above model, which will be used to choose subsequent
Revised: Q t , we will infer unknown quantities in the above model, which will be used to choose subsequent

Original: LFs λ j to query by constructing an acquisition function ϕ t :
Revised: LFs λ j to query by constructing an acquisition function ϕ t :

Original: L → R and optimizing it over λ ∈ L .
Revised: L → R and optimizing it over λ ∈ L .

Original: Acquisition Strategy and Final Set of LFs To derive an online procedure for our expert queries
Revised: Acquisition Strategy and Final Set of LFs To derive an online procedure for our expert queries

Original: we need to deﬁne the properties of the ideal subset of generated LFs L ∗ ⊂ L which we want to select.
Revised: we need to deﬁne the properties of the ideal subset of generated LFs L ∗ ⊂ L which we want to select.

Original: Prior data programming work of Ratner et al.
Revised: Prior data programming work of Ratner et al.

Original: (2016; 2019; 2020) with label models as in Eq.
Revised: (2016; 2019; 2020) with label models as in Eq.

Original: (1) does not provide an explicit analysis of ideal metrics of LF sets and their trade-offs to help deﬁne this set.
Revised: (1) does not provide an explicit analysis of ideal metrics of LF sets and their trade-offs to help deﬁne this set.

Original: We provide the following theorem, which will motivate our deﬁnition for L ∗ .
Revised: We provide the following theorem, which will motivate our deﬁnition for L ∗ .

Original: Theorem 3.1.
Revised: Theorem 3.1.

Original: Assume a binary classiﬁcation setting, m independent labeling functions with accuracy α j ∈ [0 , 1] and labeling propensity l j ∈ [0 , 1] .
Revised: Assume a binary classiﬁcation setting, m independent labeling functions with accuracy α j ∈ [0 , 1] and labeling propensity l j ∈ [0 , 1] .

Original: For a label model as in Eq.
Revised: For a label model as in Eq.

Original: (1) with given label model parameters ˆ θ ∈ R 2 m , and for any i ∈ { 1 , . . . , n } ,
Revised: (1) with given label model parameters ˆ θ ∈ R 2 m , and for any i ∈ { 1 , . . . , n } ,

Original: [Equation]
Revised: [Equation]

Original: where ˆ θ (1) are the m weights of φ Acc , and ˆ y i ∈ {− 1 , 1 } is the label model estimate for y ∗ i .
Revised: where ˆ θ (1) are the m weights of φ Acc , and ˆ y i ∈ {− 1 , 1 } is the label model estimate for y ∗ i .

Original: Proof.
Revised: Proof.

Original: The proof is given in Appendix B.
Revised: The proof is given in Appendix C.

Original: This theorem indicates that one can rank LFs according to (2 α j − 1)ˆ l j where α j , ˆ l j are the unknown accuracy and observed coverage of LF j , respectively (we provide additional analysis in Appendix B).
Revised: This theorem indicates that one can rank LFs according to (2 α j − 1) ˆ l j where α j , ˆ l j are the unknown accuracy and observed coverage of LF j , respectively (we provide additional analysis in Appendix C).

Original: Our analysis further suggests the importance of obtaining LFs with an accuracy gap above chance.
Revised: Our analysis further suggests the importance of obtaining LFs with an accuracy gap above chance.

Original: Intuitively, we do not want to add excessive noise by including LFs too close to random.
Revised: Intuitively, we do not want to add excessive noise by including LFs too close to random.

Original: Below, we assume that our ﬁnal set of LFs is sufﬁcient to accurately learn label model parameters ˆ θ , and leave analysis of the inﬂuence of additional LF properties on learning ˆ θ to future work.
Revised: Below, we assume that our ﬁnal set of LFs is sufﬁcient to accurately learn label model parameters ˆ θ , and leave analysis of the inﬂuence of additional LF properties on learning ˆ θ to future work.

Original: To deﬁne the ideal ﬁnal subset of LFs, we distinguish three scenarios: (A) there are no restrictions on the size of the ﬁnal set and any LF can be included, (B) the ﬁnal set is limited in size (e.g. due to computational considerations) but any LF can be included, (C) only LFs inspected and validated by experts may be included, e.g. due to security or legal considerations.
Revised: To deﬁne the ideal ﬁnal subset of LFs, we distinguish three scenarios: (A) there are no restrictions on the size of the ﬁnal set and any LF can be included, (B) the ﬁnal set is limited in size (e.g. due to computational considerations) but any LF can be included, (C) only LFs inspected and validated by experts may be included, e.g. due to security or legal considerations.

Original: We ﬁrst provide full details of our IWS algorithm in scenario (A).
Revised: We ﬁrst provide full details of our IWS algorithm in scenario (A).

Original: Scenario (A): Unbounded LF Set.
Revised: Scenario (A): Unbounded LF Set.

Original: When there are no restrictions as described above, our analysis in
Revised: When there are no restrictions as described above, our analysis in

Original: Appendix B indicates that the ideal subset of LFs L ∗ includes all those with accuracy greater than a gap above chance, i.e. α j > r > 0 .
Revised: Appendix C indicates that the ideal subset of LFs L ∗ includes all those with accuracy greater than a gap above chance, i.e. α j > r > 0 .

Original: Thus, we deﬁne the optimal subset in this scenario as
Revised: Thus, we deﬁne the optimal subset in this scenario as

Original: [Equation]
Revised: [Equation]

Original: This is a variation of the task of active level set estimation (LSE), where the goal is to identify all elements in a superlevel set of L (Zanette et al., 2018; Gotovos, 2013; Bryan et al., 2006).
Revised: This is a variation of the task of active level set estimation (LSE), where the goal is to identify all elements in a superlevel set of L (Zanette et al., 2018; Gotovos, 2013; Bryan et al., 2006).

Original: At each step t , we will use the straddle acquisition function (Bryan et al., 2006) for LSE, deﬁned for a candidate λ j ∈ L \ Q t − 1 to score LFs highest that are unknown and near the boundary threshold r :
Revised: At each step t , we will use the straddle acquisition function (Bryan et al., 2006) for LSE, deﬁned for a candidate λ j ∈ L \ Q t − 1 to score LFs highest that are unknown and near the boundary threshold r :

Original: [Equation]
Revised: [Equation]

Original: where σ j ( Q t − 1 )
Revised: where σ j ( Q t − 1 )

Original: = (cid:112) Var [ p ( α j | Q t − 1 )] is the standard deviation and µ j ( Q t − 1 ) =
Revised: = (cid:112) Var [ p ( α j | Q t − 1 )] is the standard deviation and µ j ( Q t − 1 ) =

Original: E [ p ( α j | Q t − 1 )] the mean of the posterior LF accuracy.
Revised: E [ p ( α j | Q t − 1 )] the mean of the posterior LF accuracy.

Original: At the end of Sec.
Revised: At the end of Sec.

Original: 3.2 we describe how we perform approximate inference of p ( α j | Q t − 1 ) via an ensemble model.
Revised: 3.2 we describe how we perform approximate inference of p ( α j | Q t − 1 ) via an ensemble model.

Original: We maximize the acquisition function over the set of candidate LFs, i.e. compute λ t = arg max λ ∈L\ Q t − 1 ϕ t ( λ ) by iterating over all unqueried λ ∈ L .
Revised: We maximize the acquisition function over the set of candidate LFs, i.e. compute λ t = arg max λ ∈L\ Q t − 1 ϕ t ( λ ) by iterating over all unqueried λ ∈ L .

Original: We then query a human expert to obtain ( λ t , u t ) , and ﬁnally update the query dataset
Revised: We then query a human expert to obtain ( λ t , u t ) , and ﬁnally update the query dataset

Original: Q t =
Revised: Q t =

Original: Q t − 1 ∪ { ( λ t , u t ) } .
Revised: Q t − 1 ∪ { ( λ t , u t ) } .

Original: After a sequence of T queries we return the following estimate of L ∗ :
Revised: After a sequence of T queries we return the following estimate of L ∗ :

Original: [Equation]
Revised: [Equation]

Original: The LF output matrix Λ , comprised of all λ j ∈ ˆ L , is then used to produce an estimate (cid:98) Y of the true class labels via the label model P θ ( Y | Λ) .
Revised: The LF output matrix Λ , comprised of all λ j ∈ ˆ L , is then used to produce an estimate (cid:98) Y of the true class labels via the label model P θ ( Y | Λ) .

Original: Finally, a noise-aware discriminative end classiﬁer f is trained on ( X, (cid:98) Y ) .
Revised: Finally, a noise-aware discriminative end classiﬁer f is trained on ( X, (cid:98) Y ) .

Original: See Algorithm 1 for pseudocode describing this full IWS-LSE procedure.
Revised: See Algorithm 1 for pseudocode describing this full IWS-LSE procedure.

Original: In our experiments, we set r = 0 .
Revised: In our experiments, we set r = 0 .

Original: 7 , though an ablation study shows that IWS-LSE works well for a range of thresholds r > 0 .
Revised: 7 , though an ablation study shows that IWS-LSE works well for a range of thresholds r > 0 .

Original: 5 (Appendix A.3, Fig.7).
Revised: 5 (Appendix B.4, Fig.

Original: Note that the LSE acquisition function aims to reduce uncertainty around r , and therefore tends to explore LFs that have coverage on parts of Y that we are still uncertain about.
Revised: Note that the LSE acquisition function aims to reduce uncertainty around r , and therefore tends to explore LFs that have coverage on parts of Y that we are still uncertain about.

Original: We denote the algorithm for scenario (A) by IWS-LSE-a .
Revised: We denote the algorithm for scenario (A) by IWS-LSE-a .

Original: Additional Cases Scenario (B): Bounded LF Set.
Revised: Additional Cases Scenario (B): Bounded LF Set.

Original: If the ﬁnal set is restricted in size to m LFs, e.g. due to computational or latency considerations when learning the label model (Eq. (1)), we need to take the trade-off of LF accuracy and LF coverage into account.
Revised: If the ﬁnal set is restricted in size to m LFs, e.g. due to computational or latency considerations when learning the label model (Eq. (1)), we need to take the trade-off of LF accuracy and LF coverage into account.

Original: Let ˆ l j be the observed empirical coverage of LF λ j .
Revised: Let ˆ l j be the observed empirical coverage of LF λ j .

Original: We want to identify LFs with accuracy above r and rank them according to their accuracy-coverage trade-off, thus our analysis in the appendix suggests the optimal subset is
Revised: We want to identify LFs with accuracy above r and rank them according to their accuracy-coverage trade-off, thus our analysis in the appendix suggests the optimal subset is

Original: [Equation]
Revised: [Equation]

Original: Since the LF accuracy-coverage trade-off only comes into effect if α j > r , we use the same acquisition function ϕ LSE t as in scenario (A), and then select the ﬁnal set as ˆ L = { λ j ∈ D :
Revised: Since the LF accuracy-coverage trade-off only comes into effect if α j > r , we use the same acquisition function ϕ LSE t as in scenario (A), and then select the ﬁnal set as ˆ L = { λ j ∈ D :

Original: [Equation]
Revised: [Equation]

Revised: 2 ∗ µ

Original: [Equation]
Revised: [Equation]

Original: µ j
Revised: Q

Original: [Equation]
Revised: [Equation]

Original: ) − 1) ∗ ˆ l j ) } which simply thresholdingby µ j ( Q T ) and then sorting.
Revised: − 1) ∗ ˆ l j ) } which corresponds to a simple thresholding and sorting operation.

Original: We denote the algorithm for scenario (B) by IWS-LSE-ac .
Revised: We denote the algorithm for scenario (B) by IWS-LSE-ac .

Original: Scenario (C): Validated LF Set.
Revised: Scenario (C): Validated LF Set.

Original: Finally, in some application scenarios only LFs inspected and validated by experts should be used to estimate Y ∗ , e.g. due to security or legal considerations.
Revised: Finally, in some application scenarios only LFs inspected and validated by experts should be used to estimate Y ∗ , e.g. due to security or legal considerations.

Original:  This leads to an active search problem (Garnett et al., 2015) where our aim is to identify a maximum number of useful LFs (i.e. u = 1 ) in L given a budget of T user queries, i.e. to compute
Revised: An LF j is validated if it is shown to an expert who then responds with u j = 1 . This leads to an active search problem (Garnett et al., 2015) where our aim is to identify a maximum number of validated

Original: L ∗ AS = arg max D⊂L , |D| =  T (cid:88) λ [Equation] ∈D
Revised: LFs (i.e. u = 1 ) in L given a budget of T user queries, i.e. to compute

Original: [Equation]
Revised: [Equation]

Original:  As in (Garnett et al., 2015; Jiang et al., 2017), we use a one-step look ahead active search acquisition function deﬁned for a candidate λ j ∈ L \ Q t − 1 to be the posterior probability that the usefulness label u j is positive, i.e. ϕ AS t ( λ j ) = µ j ( Q t − 1 ) .
Revised: j = 1 } . As in (Garnett et al., 2015; Jiang et al., 2017), we use a one-step look ahead active search acquisition function deﬁned for a candidate λ j ∈ L \ Q t − 1 to be the posterior probability that the usefulness label u j is positive, i.e. ϕ AS t ( λ j ) = µ j ( Q t − 1 ) .

Original: We denote the algorithm for scenario (C) by IWS-AS .
Revised: We denote the algorithm for scenario (C) by IWS-AS .

Original: (IWS-LSE-a). Input : L : set of LFs, T : max iterations, p ( u | λ, Q ) : usefulness model for ensemble, ϕ t : acquisition function.
Revised: Input : L : set of LFs, T : max iterations.

Original: 1 Q 0 ← ∅ 2 for t = 1 , 2 , . .
Revised: 1 Q 0 ← ∅ 2 for t = 1 , 2 , . .

Original: , T do λ t ← arg max λ
Revised: , T do λ t ← arg max λ

Original: ∈L\ Q t − 1 ϕ t ( λ ) (cid:46) Eq.
Revised: ∈L\ Q t − 1 ϕ t ( λ ) (cid:46) Eq.

Original: (4) u t ← ExpertQuery
Revised: (4) u t ← ExpertQuery

Original: ( λ t )
Revised: ( λ t )

Original: Q t ← Q t − 1 ∪ { ( λ t , u t ) } 6 end 7 ˆ L ← { λ j ∈ L : E [ p ( α j | Q T )] > r } (cid:46) Eq.
Revised: Q t ← Q t − 1 ∪ { ( λ t , u t ) } 6 end 7 ˆ L ← { λ j ∈ L : E [ p ( α j | Q T )] > r } (cid:46) Eq.

Original: Approximate Inference Details We now describe how we use our expert-feedback model in Eq.
Revised: Approximate Inference Details We now describe how we use our expert-feedback model in Eq.

Original: (2) to infer p ( α j | Q t ) .
Revised: (2) to infer p ( α j | Q t ) , which is necessary for the acquisition functions and ﬁnal set estimates.

Original: We deﬁned a generative model of human feedback u j on query LF λ
Revised: Recall that we deﬁned a generative model of human feedback u j on query LF λ

Original: j with latent variables v j and ω .
Revised: j with latent variables v j and ω .

Original: We assumed a connection between v j and the latent LF accuracy α j via a monotonic increasing function α j = g ( v j ) .
Revised: We assumed a connection between v j and the latent LF accuracy α j via a monotonic increasing function α j = g ( v j ) .

Original: Similar to existing work on high dimensional uncertainty estimation (Beluch et al., 2018; Chitta et al., 2018), we use an ensemble { ˜ h ω ( i ) } si = of s neural networks ˜ h ω with parameters ω to predict u j given input λ j .
Revised: Similar to existing work on high dimensional uncertainty estimation (Beluch et al., 2018; Chitta et al., 2018), we use an ensemble { ˜ h ω ( i ) } si =1 of s neural networks ˜ h ω with parameters ω to predict u j given input λ j .

Original: To perform this prediction, we need a feature representation τ ( λ j ) for LFs that is general and works for any data type and task.
Revised: To perform this prediction, we need a feature representation τ ( λ j ) for LFs that is general and works for any data type and task.

Original: For these features, we use the LF output over our unlabeled dataset, i.e. τ 0 ( λ j ) =
Revised: For these features, we use the LF output over our unlabeled dataset, i.e. τ 0 ( λ j ) =

Original: ( λ j ( x 1 ) , . . .
Revised: ( λ j ( x 1 ) , . . .

Original: , λ j ( x n )) .
Revised: , λ j ( x n )) .

Original: We then project this to d (cid:48) dimensions using PCA for a ﬁnal feature representation τ ( λ j ) , which is given as input to each ˜ h ω .
Revised: We then project this to d (cid:48) dimensions using PCA for a ﬁnal feature representation τ ( λ j ) , which is given as input to each ˜ h ω .

Original: Our neural network ensemble can then effectively learn functions ˜
Revised: Our neural network ensemble can then effectively learn functions ˜

Original: h : R d (cid:48)
Revised: h : R d (cid:48)

Original: → [0 , 1] , which map from features τ ( λ j ) to v j = p ( u j = 1 | Q t ) .
Revised: → [0 , 1] , which map from features τ ( λ j ) to v j = p ( u j = 1 | Q t ) .

Original: This yields an ensemble of estimates for v j , and through g − 1 , of α j .
Revised: This yields an ensemble of estimates for v j , and through g − 1 , of α j .

Original: These are treated as approximate samples from p ( α j | Q t ) , and used to form sample-estimates for the above acquisition functions.
Revised: These are treated as approximate samples from p ( α j | Q t ) , and used to form sample-estimates for the above acquisition functions.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: Our experiments show that heuristics obtained via a small number of iterations of IWS can be used to train a downstream end classiﬁer f with highly competitive test set performance.
Revised: Our experiments show that heuristics obtained via a small number of iterations of IWS can be used to train a downstream end classiﬁer f with highly competitive test set performance.

Original: We ﬁrst present results obtained with a simulated noiseless IWS oracle.
Revised: We ﬁrst present results obtained with a simulated noiseless IWS oracle.

Original: Oracle experiments allow us to answer how our method would perform if users had perfect knowledge about LF accuracy.
Revised: Oracle experiments allow us to answer how our method would perform if users had perfect knowledge about LF accuracy.

Original: We then show results from a real-user study, involving experiments in which the query feedback is given by humans.
Revised: We then show results from a real-user study, involving experiments in which the query feedback is given by humans.

Revised: We provide extended dataset and implementation details in Appendix B.

Original:  We created six binarytext classiﬁcation tasks on three publicly available datasets:Amazon reviews,
Revised: Datasest Text Classiﬁcation: We created six binary tasks on three publicly available datasets:

Original:  IMDB reviews, and BiasBios biographies.
Revised: Amazon reviews, IMDB reviews, and BiasBios biographies.

Original: We provide full dataset and implementation details in
Revised: The tasks were chosen such that most

Original: Appendix A.
Revised: [Figure]

Original: The tasks were chosen such that most English speakers can provide sensible expert feedback on LFs, for ease of reproducibility.
Revised: English speakers can provide sensible expert feedback on LFs, for ease of reproducibility.

Original: Similar text classiﬁcation datasets have been used in related work (Varma & R´e, 2018; Varma et al., 2019).
Revised: Similar text classiﬁcation datasets have been used in related work (Varma & R´e, 2018; Varma et al., 2019).

Original: We note that a head to head comparison of user effort required to respond to sample annotation queries vs LF usefulness queries is application dependent.
Revised: Multi-modal classiﬁcation: As in (Varma & R´e, 2018), we take the COCO dataset (Lin et al., 2014) and generate LFs over captions, while classiﬁcation is performed on the associated images. The two binary tasks are to identify a ‘person’ in an image, and to identify ‘sports’ in an image.

Original:  We do however provide a timing study to showcase the effort required to carry out IWS versus labeling of samples in our speciﬁc user experiments, see Table 1.
Revised: Image classiﬁcation: We use the COCO dataset and generate nearest-neighbor based heuristics directly on the images. The binary classiﬁcation tasks are to identify ‘sports’ in an image and ‘vehicle’ in an image.

Original: Approaches All approaches train the same downstream end classiﬁer f one the same inputs X .
Revised: Approaches All approaches train the same downstream end classiﬁer f one the same inputs X .

Original: We show results for IWS-LSE-a (unbounded LF set), IWS-LSE-ac (bounded LF set), and IWS-AS (validated LF set).
Revised: We show results for IWS-LSE-a (unbounded LF set), IWS-LSE-ac (bounded LF set), and IWS-AS (validated LF set).

Revised: For IWS-LSE-ac , we bound the size of the ﬁnal set of LFs at each iteration t by m = (cid:80) t − 1 i =1 u

Revised: i + ˜ m , i.e. the number of LFs so far annotated as u = 1 plus a constant ˜ m . .

Original: We compare the test set performance of IWS to a set of alternatives including (1) annotation of samples via active learning (uncertainty sampling) by a noiseless oracle, (2) the Snuba system (Varma & R´e, 2018) 1 , and (3) the performance obtained using all ground truth training labels .
Revised: We compare the test set performance of IWS to a set of alternatives including (1) annotation of samples via active learning (uncertainty sampling) by a noiseless oracle, (2) the Snuba system (Varma & R´e, 2018), and (3) the performance obtained using all ground truth training labels .

Original: In Appendix A we also compare our results to the IWS procedure using a random acquisition function ( IWS-random ), in the validated LF set setting.
Revised: In Appendix B we provide additional results comparing IWS to itself, but with a random acquisition function ( IWS-random ). In our ﬁgures, annotations on the x-axis correspond to labeled samples for Snuba and active learning, and to labeled LFs for IWS. We note that this head to head comparison of user effort is application dependent. We provide a timing study to showcase the effort required to carry out IWS versus labeling of samples in our speciﬁc user experiments, see Table 1.

Original: Creating LF Families for Text Related work (Ratner et al., 2020; Varma et al., 2019) previously demonstrated that word and phrase LFs can provide good weak supervision sources on text.
Revised: LF Families For text, Ratner et al. ; Varma et al. (2019) demonstrate that word and phrase LFs can provide good weak supervision sources for text tasks.

Original: We deﬁne[Figure] a uni-gram vocabulary over all documents and discardvery high and low frequency terms.
Revised: To generate LFs, we deﬁne a uni-gram vocabulary over all documents and discard high and low frequency terms.

Original: We then exhaustively generate LFs from an LF family z φ which outputs a target label if a uni-gram appears in a document, where φ speciﬁes the uni-gram and target label.
Revised: We then exhaustively generate LFs from an LF family z φ which outputs a target label if a uni-gram appears in a document, where φ speciﬁes the uni-gram and target label.

Original: We also evaluated combinations of higher-order n-grams
Revised: We also evaluated combinations of higher-order n-grams, but did not observe a signiﬁcant change in performance.

Original: [Figure]
Revised: For COCO images, it is difﬁcult to obtain strong domain primitives to create weak supervision sources, even from scratch. We hence choose the images and their embeddings themselves to do this job for us, by relying on k-nearest neighbor functions. To generate LFs with high coverage, we ﬁrst create small, unique clusters of up to k 1 mutual nearest neighbors (MkNN) 1 . For each member of a cluster, we then ﬁnd the knearest neighbors, and keep ones shared by at least one other cluster member.

Original: but did not observe a signiﬁcant change in performance
Revised: Finally, each extended cluster deﬁnes an LF, which assigns the same label to each member of the extended cluster. The

Original: [Figure]
Revised: , and image B MkNN symmetry produces good initial clusters of varying size, while the second produces LFs with large and varying coverage. ﬁnd that whether an LF is predictive of the latent class variable at better than random accuracy can be quickly assessed by a user by visually inspecting the consistency of the initial cluster and a small selection of the cluster nearest neighbors.

Original: 4. O RACLE EXPERIMENTS
Revised: 4. O RACLE EXPERIMENTS

Original: Our oracle labels an LF as useful if it has an accuracy of at least 0 .
Revised: Our oracle labels an LF as useful if it has an accuracy of at least 0 .

Original: Measured by test-set AUC of ﬁnal classiﬁer f , IWS-LSE outperformsall other approaches signiﬁcantly on ﬁve out of six  datasets, and matches the best performance also attained by Snuba on one dataset (Fig. 2).
Revised: Measured by test-set AUC of ﬁnal classiﬁer f , IWS-LSE outperforms other approaches signiﬁcantly on ﬁve out of six text datasets, and matches the best performance also attained by Snuba on one dataset (Fig. 2).

Original: IWS-AS (Fig. 4,
Revised: IWS-AS (Fig. 5,

Original: Appendix) performs similarly well on four  datasets, and competitively on the other two.
Revised: Appendix) performs similarly well on four text datasets, and competitively on the other two.

Original: Both IWS approaches outperform active learning by a wide margin on all  datasets.
Revised: Both IWS approaches outperform active learning by a wide margin on all text datasets.

Original: IWS also quickly approaches the performance achieved by an end model trained on the full ground truth training labels.
Revised: IWS also quickly approaches the performance achieved by an end model trained on the full ground truth training labels.

Original: We provide IWS-LSE annealing results varying the ﬁnal set size as well as thresholds r inAppendix A.
Revised: We provide IWS-LSE annealing results varying the ﬁnal set size as well as thresholds r in

Revised: Appendix B. For the COCO image tasks, LFs were created using image captions as in (Varma & R ´ e, 2018)

Revised: (Fig. 4, ﬁrst and second plot), as well as on images directly via nearest neighbors (Fig. 4, third and fourth plot).

Revised: IWS continues to perform competitively, and quickly approaches the performance achieved using all training ground truth.

Original: 4. U SER EXPERIMENTS
Revised: 4. U SER EXPERIMENTS

Original: [Table]
Revised: [Table]

Original: We conduct experiments of IWS-AS with real users on the Amazon and IMDB review sentiment classiﬁcation tasks.
Revised: We conduct experiments of IWS-AS with real users on the Amazon and IMDB review sentiment classiﬁcation tasks.

Original: The results demonstrate that users judge high accuracy functions as useful and make few mistakes.
Revised: The results demonstrate that users judge high accuracy functions as useful and make few mistakes.

Original: In our experiments, users are shown a description of the heuristic (the key term pattern) and the intended label.
Revised: In our experiments, users are shown a description of the heuristic (the key term pattern) and the intended label.

Original: Users are also shown four snippets of random documents where the LF applied, but are instructed to ﬁrst consider the LF without inspecting these random samples, and to only consider the examples if necessary.
Revised: Users are also shown four snippets of random documents where the LF applied, but are instructed to ﬁrst consider the LF without inspecting these random samples, and to only consider the examples if necessary.

Original: See Appendix A.2 for a screenshot of the query interface and more details regarding the user prompts.
Revised: See Appendix B.2 for a screenshot of the query interface and more details regarding the user prompts.

Original: The top of Fig.
Revised: The top of Fig.

Original: 3 shows that mean test set performance of IWS-AS using LFs obtained from real user feedback closely tracks the simulated oracle performance after about 100 iterations.
Revised: 3 shows that mean test set performance of IWS-AS using LFs obtained from real user feedback closely tracks the simulated oracle performance after about 100 iterations.

Original: Fig.
Revised: Fig.

Original: 3 further shows the queried LFs and  corresponding user responses by their true accuracy vs. their non-abstain votes.
Revised: 3 further shows the queried LFs and [Figure] corresponding user responses by their true accuracy vs. their non-abstain votes.

Original: To match the mean test AUC of IWS-AS obtained after 200 iterations, active learning (uncertainty sampling) requires about 600 iterations.
Revised: To match the mean test AUC of IWS-AS obtained after 200 iterations, active learning (uncertainty sampling) requires about 600 iterations.

Original: For the IMDB dataset, to achieve the same mean test AUC of IWS-AS obtained after 200 iterations, active learning requires more than 1000 iterations.
Revised: For the IMDB dataset, to achieve the same mean test AUC of IWS-AS obtained after 200 iterations, active learning requires more than 1000 iterations.

Original: For both datasets, the average response time to each query was fast.
Revised: For both datasets, the average response time to each query was fast.

Original: We conducted a manual labeling exercise of samples for the
Revised: We conducted a manual labeling exercise of samples for the

Original: IMDB and Amazon datasets (Table 1) with real users.
Revised: IMDB and Amazon datasets (Table 1) with real users.

Original: Users incorrectly classiﬁed 9% of IMDB reviews while taking signiﬁcantly longer compared to the response times to LF queries.
Revised: Users incorrectly classiﬁed 9% of IMDB reviews while taking signiﬁcantly longer compared to the response times to LF queries.

Original: For the
Revised: For the

Original: Amazon dataset, users mislabeled 2% of samples and were also slower at labeling samples than LFs.
Revised: Amazon dataset, users mislabeled 2% of samples and were also slower at labeling samples than LFs.

Original: The user-study experiments involved nine persons with a computer science background.
Revised: The user-study experiments involved nine persons with a computer science background.

Original: Neither the true accuracy of each heuristic nor the end model train or test set results were revealed to the users at any stage of the experiment.
Revised: Neither the true accuracy of each heuristic nor the end model train or test set results were revealed to the users at any stage of the experiment.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We have proposed methods for interactively discovering weak supervision sources.
Revised: We have proposed methods for interactively discovering weak supervision sources.

Original: Our results show that a small number of expert interactions can sufﬁce to select good weak supervision sources from a large pool of candidates, leading to competitive end classiﬁers.
Revised: Our results show that a small number of expert interactions can sufﬁce to select good weak supervision sources from a large pool of candidates, leading to competitive end classiﬁers.

Original: The proposed methodology shows promise as a way to signiﬁcantly speed up the process of weak supervision source deﬁnition by domain experts as an alternative to devising such sources from scratch.
Revised: The proposed methodology shows promise as a way to signiﬁcantly speed up the process of weak supervision source deﬁnition by domain experts as an alternative to devising such sources from scratch.

Original: We obtain superior predictive performance ona downstream test set compared to the automatic selection of LFs with Snuba (Varma & R ´ e, 2018) and standard active learning (where users annotate samples instead of LFs), when measured with respect to the number of user queries.
Revised: On a large number of tasks, we obtain superior predictive performance on downstream test sets compared to the automatic selection of LFs with Snuba (Varma & R´e, 2018) and standard active learning (where users annotate samples instead of LFs), when measured with respect to the number of user annotations.

Original: We conduct experiments with real users on two popular benchmark datasets and ﬁnd that humans recognize and approve high accuracy LFs, yielding models that match performance attainable with a simulated oracle.
Revised: We conduct experiments with real users on two popular benchmark datasets and ﬁnd that humans recognize and approve high accuracy LFs, yielding models that match performance attainable with a simulated oracle.

Original: Further, in our experiments we ﬁnd that users annotate heuristics faster than individual samples on the datasets we used.
Revised: Further, in our experiments we ﬁnd that users annotate heuristics faster than individual samples on the datasets we used.

Original: There are limitations to the approaches we propose.
Revised: There are limitations to the approaches we propose.

Original: LF families that generate heuristics (such as  shallow decision trees) appear easy to deﬁne for many text classiﬁcation tasks, and are reusable thereafter for similar tasks and datasets.
Revised: LF families that generate heuristics—such as keyword functions or shallow decision trees—appear straightforward to deﬁne for some text classiﬁcation tasks, and are reusable thereafter for similar tasks and datasets.

Original: However,in future work we need to show that this process works similarly well for other data types and tasks.
Revised: However, future work is required to show that this process works similarly well for other data types and tasks.

Original: Furthermore, we rely on experts to be able to judge heuristics pre-generated from an LF family in the same way experts  judge heuristics they create from scratch, which we cannot assume will be possible for all LF families.
Revised: Additionally, we rely on an expert’s ability to judge LFs generated from an LF family. While experts are similarly required to judge the quality of LFs they create from scratch, this level of interpretability may be difﬁcult to achieve across generated LFs for some tasks and data types. Also, in some cases the annotation of LFs could be considerably slower than the annotation of samples. Furthermore, datasets with a large number of classes may lead to very large sets of candidate LFs.

Original: The approach proposed in this paper is not meant to replace active learning or manually created weak supervision sources.
Revised: Thus, future work will need to ﬁnd smart and efﬁcient ways to traverse this space or to generate LF candidates on the ﬂy. Future work should also explore hybrid methods of IWS and active learning, interactive learning of LF dependencies, novel types of LF families, and acquisition functions which optimize for additional properties of sets of heuristics such as diversity. Our proposed approach is not meant to replace active learning or manually created weak supervision.

Original: For manyimportant tasks, standard active learning or crowd sourcing may be entirely sufﬁcient.
Revised: For many tasks, standard active learning, data programming, or crowd-sourcing may be entirely sufﬁcient.

Original: However, when a large number of labels is necessary to train good models while experts’ effort is precious, an interactive framework for weak supervision can be the right choice. 
Revised: However, when a large number of labels is necessary to train good models while experts’ effort is precious, an interactive framework for weak supervision can be the right choice. A P SEUDOCODE FOR DIFFERENT IWS SCENARIOS sion (IWS-LSE-ac).

Original: Promising directions for future work are to explore hybrid methods of querying for heuristics and labels for samples, interactive learning of dependencies, and acquisition functions which optimize for additional properties of sets of heuristics such as LF diversity.
Revised: Input : L : set of LFs, T : max iterations. 1 Q 0 ← ∅ 2 for t = 1 , 2 , . . , T do λ t ← arg max λ ∈L\ Q t − 1 ϕ t ( λ ) (cid:46) Eq.

Revised: (4) u t ← ExpertQuery

Revised: ( λ t )

Revised: Q t ← Q t − 1 ∪ { ( λ t , u t ) } 6 end 7 ˆ L ← { λ j ∈ D : arg max D⊆L , |D| = m (cid:80) λ

Revised: j ∈D ( 1 { µ j ( Q T ) >r } (2 ∗ µ j ( Q T ) − 1) ∗ ˆ l j ) }

Revised: Input : L : set of LFs, T : max iterations.

Revised: 1 Q 0 ← ∅ 2 for t = 1 , 2 , . .

Revised: , T do λ t ← arg max λ

Revised: ∈L\ Q t − 1 µ ( Q t − 1 ) u t ← ExpertQuery

Revised: ( λ t )

Revised: Q t ← Q t − 1 ∪ { ( λ t , u t ) } 6 end 7 ˆ L ← { λ j ∈ Q T : u j = 1 }

Revised: Here we provide pseudocode for the IWS-AS and IWS-LSE-ac settings, while the procedure for

Revised: IWS-LSE-a can be found in the main paper in Algorithm 1.

Revised: Let us recap why we arrive at different formulations for IWS.

Revised: In Sec. 3.2, we distinguish three scenarios for arriving at a ﬁnal set of weak supervision sources which are modeled to obtain an estimate of the latent class variable Y .

Revised: All three scenarios lead to different deﬁnitions of an optimal ﬁnal set of LFs, which in turn means that they require us to formulate appropriate acquisition functions to achieve a good estimate of the optimal set within a budget of T expert interactions.

Revised: In scenario (A), we place no restrictions on the size of the ﬁnal set and any LF can be included in it.

Revised: This means that we have the computational resources to model a potentially very large number of weak supervision sources, and we do not require domain experts to inspect and validate every single LF that is modeled.

Revised: Importantly, this means that we can and should include LFs that are good according to our predictive model and our deﬁnition of the optimal ﬁnal set of LFs, but have never been shown to an expert.

Revised: In scenario (B), the ﬁnal set is limited in size but LFs do not have to be inspected and validated by a user.

Revised: This scenario may be attractive for rapid cycles during development when a very large number of LFs becomes computationally prohibitive.

Revised: Finally, in scenario (C), only LFs inspected and validated by experts may be included, e.g. due to security or legal considerations.

Original: A A DDITIONAL D ETAILS ON E XPERIMENTS AND R ESULTS
Revised: B A DDITIONAL D ETAILS ON E XPERIMENTS AND R ESULTS

Original: Datasets We use three publicly available datasets 2 and deﬁne six binary text classiﬁcation tasks.
Revised: Datasets For our text data experiments, we use three publicly available datasets 2 to deﬁne six binary text classifaction tasks.

Original: We use a subset of the Amazon Review Data (He & McAuley, 2016) for sentiment classiﬁcation, aggregating all categories with more than 100 k reviews from which we sample 200 k reviews and split them into 160 k training points and 40 k test points.
Revised: We use a subset of the Amazon Review Data (He & McAuley, 2016) for sentiment classiﬁcation, aggregating all categories with more than 100 k reviews from which we sample 200 k reviews and split them into 160 k training points and 40 k test points.

Original: We use the IMDB Movie Review Sentiment dataset (Maas et al., 2011) which has 25 k training samples and 25 k test samples.
Revised: We use the IMDB Movie Review Sentiment dataset (Maas et al., 2011) which has 25 k training samples and 25 k test samples.

Original: In addition, we use the Bias in Bios (De-Arteaga et al., 2019) dataset from which we create binary classiﬁcation tasks to distinguish difﬁcult pairs among frequently occurring occupations.
Revised: In addition, we use the Bias in Bios (De-Arteaga et al., 2019) dataset from which we create binary classiﬁcation tasks to distinguish difﬁcult pairs among frequently occurring occupations.

Original: Speciﬁcally, we create the following subsets with equally sized train and test sets: journalist or photographer ( n = 32 258 ), professor or teacher ( n = 24 588 ), painter or architect ( n = 12 236 ), professor or physician ( n = 54 476 ). 
Revised: Speciﬁcally, we create the following subsets with equally sized train and test sets: journalist or photographer ( n = 32 258 ), professor or teacher ( n = 24 588 ), painter or architect ( n = 12 236 ), professor or physician ( n = 54 476 ). For the multi-modal tasks of text captions and images as well as the pure image task we use the COCO dataset (Lin et al., 2014). We take the ofﬁcial validation set ( n = 4952 ) as the test set.

Original:  Implementation Details Our probabilistic ensemble in IWS, which is used in all acquisition functions to learn p ( u j = 1 | Q t ) , is a bagging ensemble of s = 50 multilayer perceptrons with two hidden layers of size 10, RELU activations, sigmoid output and logarithmic loss.
Revised: This set of images is never used at any other point in the pipeline. Implementation Details Our probabilistic ensemble in IWS, which is used in all acquisition functions to learn p ( u j = 1 | Q t ) , is a bagging ensemble of s = 50 multilayer perceptrons with two hidden layers of size 10, RELU activations, sigmoid output and logarithmic loss.

Original: To create features for the p candidate LFs in L , we use singular value decomposition (SVD) to project from n to d (cid:48)
Revised: To create features for the p candidate LFs in L , we use singular value decomposition (SVD) to project from Amazon: [Figure] IWS

Revised: LSE and IWS-AS are n to d (cid:48)

Original: = 150 .
Revised: = 150 .

Original: Thus, at iteration t , given a query dataset Q t − 1 = { ( λ j , u j ) } t − 1 j =1 , the ensemble is trained on pairs { ( τ ( λ j ) , u j ) } t − 1 j =1 where τ ( λ j ) are the SVD features and u j the binary expert responses.
Revised: Thus, at iteration t , given a query dataset Q t − 1 = { ( λ j , u j ) } t − 1 j =1 , the ensemble is trained on pairs { ( τ ( λ j ) , u j ) } t − 1 j =1 where τ ( λ j ) are the SVD features and u j the binary expert responses.

Original: The output of the ensemble on LFs not in the query dataset is used to compute σ j ( Q t − 1 ) =
Revised: The output of the ensemble on LFs not in the query dataset is used to compute σ j ( Q t − 1 ) =

Original: (cid:112)
Revised: (cid:112)

Original: Var [ g − 1 ( p ( u j = 1 | Q t − 1 ))] and µ j ( Q t − 1 ) =
Revised: Var [ g − 1 ( p ( u j = 1 | Q t − 1 ))] and µ j ( Q t − 1 ) =

Original: E [ g − 1 ( p ( u j = 1 | Q t − 1 ))] .
Revised: E [ g − 1 ( p ( u j = 1 | Q t − 1 ))] .

Original: While g , which maps α j to v j , could be ﬁne-tuned from data, we set g as the identity function in our experiments, which we ﬁnd works well empirically.
Revised: While g , which maps α j to v j , could be ﬁne-tuned from data, we set g as the identity function in our experiments, which we ﬁnd works well empirically.

Original: Finally, to allow human experts to express some level of conﬁdence about their decision on u j , we also collect corresponding uncertainty weights b j ∈ { 1 , 0 . 5 } , and we multiply the contribution to the loss of each u j by the respective weight b j .
Revised: Finally, to allow human experts to express some level of conﬁdence about their decision on u j , we also collect corresponding uncertainty weights b j ∈ { 1 , 0 . 5 } , and we multiply the contribution to the loss of each u j by the respective weight b j .

Original: Users can also skip queries if they are unsure, indicated in black in Fig.
Revised: Users can also skip queries if they are unsure, indicated in black in Fig.

Original: These unsure responses are still counted as an iteration/query in our plots.
Revised: These unsure responses are still counted as an iteration/query in our plots.

Original: Our downstream end classiﬁer f is a multilayer perceptron with two hidden layers of size 20 and
Revised: Our downstream end classiﬁer f is a multilayer perceptron with two hidden layers of size 20 and

Original: RELU activations, sigmoid output and logarithmic loss.
Revised: RELU activations, sigmoid output and logarithmic loss.

Original: Each model in the ensemble as well as f are optimized using Adam (Kingma & Ba, 2014).
Revised: Each model in the ensemble as well as f are optimized using Adam (Kingma & Ba, 2014).

Original: To avoid conﬂating performance gains by using pre-trained representations, we ﬁt the end models f to low dimensional projections of a large bag-of-words matrix via truncated Singular Value Decomposition (SVD), ﬁxing the embedding size to d = 300 .
Revised: For the text datasets, we ﬁt the end models f to low dimensional projections of a large bag-of-words matrix via truncated Singular Value Decomposition (SVD), ﬁxing the embedding size to d = 300 .

Original: We repeat each experiment ten times.
Revised: We repeat each experiment ten times.

Original: We assume that the class balance is known when ﬁtting the label model, as common in related work.
Revised: We assume that the class balance is known when ﬁtting the label model, as common in related work.

Original: When class balance is unknown, (Ratner et al., 2019) discuss an unsupervised approach to estimate it. 
Revised: When class balance is unknown, (Ratner et al., 2019) discuss an unsupervised approach to estimate it. For the

Revised: COCO image experiments, we use the second-to-last layer of a ResNet-18 (He et al., 2016) pretrained on ImageNet to obtain image features.

Revised: These image features are used as the embedding to train the end classiﬁer for all approaches which we compare.

Revised: The embeddings are also used to create the nearest-neighbor based image LFs.

Original: The ﬁrst 8 iterations of IWS are initialized with queries of four LFs known to have accuracy between 0 . 7 and 0 .
Revised: The ﬁrst 8 iterations of IWS are initialized with queries of four LFs known to have accuracy between 0 . 7 and 0 .

Original: 75 drawn at random and four randomly drawn LFs with arbitrary accuracy.
Revised: 75 drawn at random and four randomly drawn LFs with arbitrary accuracy.

Original: Subsequently,
Revised: Subsequently,

Original: IWS chooses the next LFs to query.
Revised: IWS chooses the next LFs to query.

Original: Active learning is initialized with the same number of known samples.
Revised: Active learning is initialized with the same number of known samples.

Original: A. F ULL IWS R ESULTS AND A LL B ASELINES
Revised: B. F ULL IWS R ESULTS AND A LL B ASELINES

Original: Fig.
Revised: Fig.

Original: 4 provides the full results of all IWS settings to all baselines, including IWS with a random acquisition function (IWS-random).
Revised: 5 provides the full results of all IWS settings to all baselines, including IWS with a random acquisition function (IWS-random).

Original: IWS LSE-a corresponds to scenario (A) where there are no restrictions on the size of the ﬁnal set and any LF can be included]. 
Revised: IWS LSE-a corresponds to scenario (A) where there are no restrictions on the size of the ﬁnal set and any LF can be included]. IWS LSE-ac corresponds to scenario (B) where the ﬁnal set LFs is limited in size (e.g. due to computational considerations) but

Original: IWS
Revised: Before starting the experiment, users were provided with a description of the task and the labeling function family. any LF can be included. IWS-AS corresponds to Scenario (C), where only LFs in our query dataset Q t can be used, which are LFs that were inspected and validated by experts, e.g. due to security or legal considerations.

Original: [Figure]
Revised: [Figure]

Original: IWS-LSE and IWS-AS are compared to active learning, Snuba training on all

Original: A. U SER EXPERIMENTS A.2. I NTERFACE AND EXPERIMENT PROMPT

Original: Fig.
Revised: Fig.

Original: 5 shows an example of the prompt that was shown to users at each iteration of the IWS user experiments.
Revised: 6 shows an example of the prompt that was shown to users at each iteration of the IWS user experiments.

Original: Before the experiment started, users were ﬁrst instructed on the interface they would see and the task they would be given, i.e. to label a heuristic as good if they would expect it to label samples at better than random accuracy and as bad otherwise.
Revised: Before the experiment started, users were ﬁrst instructed on the interface they would see and the task they would be given, i.e. to label a heuristic as good if they would expect it to label samples at better than random accuracy and as bad otherwise.

Original: Users were also instructed about the response options, including the option to not answer a query if they were unsure (‘I don’t know’).
Revised: Users were also instructed about the response options, including the option to not answer a query if they were unsure (‘I don’t know’).

Original: Users were given a description of the classiﬁcation task and domain of the documents for which heuristics were being acquired.
Revised: Users were given a description of the classiﬁcation task and domain of the documents for which heuristics were being acquired.

Original: Users were also provided with a description of the heuristic generated which labeled samples with a target label if a document contained a certain term.
Revised: Users were also provided with a description of the heuristic generated which labeled samples with a target label if a document contained a certain term.

Original: Finally, users were given two examples of a better than random heuristic, and two examples of an arbitrary heuristic.
Revised: Finally, users were given two examples of a better than random heuristic, and two examples of an arbitrary heuristic.

Original: During the experiments, users were also provided with 4 random examples of documents documents where the queried LF applied.
Revised: During the experiments, users were also provided with 4 random examples of documents documents where the queried LF applied.

Original: Users were instructed to ﬁrst consider the LF without inspecting these random samples, and to only consider the examples if necessary.
Revised: Users were instructed to ﬁrst consider the LF without inspecting these random samples, and to only consider the examples if necessary.

Original: While LFs receive binary labels, users were allowed to express uncertainty about their decision, which was used as a sample weight ( 1 if certain else 0 . 5 ) of LFs during training of the probabilistic model of user feedback.
Revised: While LFs receive binary labels, users were allowed to express uncertainty about their decision, which was used as a sample weight ( 1 if certain else 0 . 5 ) of LFs during training of the probabilistic model of user feedback.

Original: A.2. A DDITIONAL STATISTICS OF USER EXPERIMENTS
Revised: B.2. A DDITIONAL STATISTICS OF USER EXPERIMENTS

Original: In Fig.
Revised: In Fig.

Original: 6 we provide more details about our user experiments.
Revised: 7 we provide more details about our user experiments.

Original: The top row displays the test set performance of downstream model f for each individual user.
Revised: The top row displays the test set performance of downstream model f for each individual user.

Original: The middle row shows how the number of LFs determined by the user to be useful u = 1 increases with the number ofiterations.
Revised: The middle row shows how the number of LFs determined by the user to be useful u = 1 increases with the number of

Original: The bottom row displays the maximum positive correlation between a new LF with u = 1 at iteration t and all previously accepted LFs with u = 1 up to iteration t . Note that we take abstains into account by computing correlation between and LF i and j only on entries where at least one of them is nonzero A.2. H EURISTICS FOUND BY IWS-AS
Revised: B. H EURISTICS FOUND B.3. IWS WITH REAL USERS , SENTIMENT CLASSIFICATION

Original: We here provide some concrete examples of heuristics found during the IWS-AS procedure, that real users annotated as useful during the experiments.
Revised: We here provide some concrete examples of heuristics found during the IWS-AS procedure, that real users annotated as useful during the experiments.

Original: For the IMDB dataset:
Revised: For the IMDB dataset:

Original: Before starting the experiment, users were provided with a description of the task and the labeling function family.

Original: • Ten terms most frequently annotated as useful by users – Positive: beautiful, wonderful, perfect, enjoyed, amazing, brilliant, fantastic, superb, excellent, masterpiece.
Revised: • Ten terms most frequently annotated as useful by users – Class=1: beautiful, wonderful, perfect, enjoyed, amazing, brilliant, fantastic, superb, excellent, masterpiece.

Original: – Negative: worst, poor, awful, bad, waste, terrible, horrible, boring, crap, stupid.
Revised: – Class=0: worst, poor, awful, bad, waste, terrible, horrible, boring, crap, stupid.

Original: • Ten terms annotated as useful with highest underlying accuracy – Positive: ﬂawless, superbly, perfection, wonderfully, captures, refreshing, breathtaking, delightful, beautifully, underrated.
Revised: • Ten terms annotated as useful with highest underlying accuracy – Class=1: ﬂawless, superbly, perfection, wonderfully, captures, refreshing, breathtaking, delightful, beautifully, underrated.

Original: – Negative: stinker, dreck, unwatchable, unfunny, waste, atrocious, pointless, redeeming, laughable, lousy.
Revised: – Class=0: stinker, dreck, unwatchable, unfunny, waste, atrocious, pointless, redeeming, laughable, lousy.

Original: • Ten terms annotated as useful by users, selected at random – Positive: favorites, joy, superbly, delight, wonderfully, art, intelligent, terriﬁc, light, ﬁnest.
Revised: • Ten terms annotated as useful by users, selected at random – Class=1: favorites, joy, superbly, delight, wonderfully, art, intelligent, terriﬁc, light, ﬁnest.

Original: – Negative: reason, failed, atrocious, decent, unfunny, lame, ridiculous, mistake, worst, dull.
Revised: – Class=0: reason, failed, atrocious, decent, unfunny, lame, ridiculous, mistake, worst, dull.

Original: For the Amazon dataset:
Revised: For the Amazon dataset:

Original: • Ten terms most frequently annotated as useful by users – Positive: wonderful, beautiful, amazing, fantastic, favorite, awesome, love, best, perfect, easy.
Revised: • Ten terms most frequently annotated as useful by users – Class=1: wonderful, beautiful, amazing, fantastic, favorite, awesome, love, best, perfect, easy.

Original: – Negative: worst, terrible, horrible, awful, worse, boring, poor, bad, waste, garbage.
Revised: – Class=0: worst, terrible, horrible, awful, worse, boring, poor, bad, waste, garbage.

Original: • Ten terms annotated as useful with highest underlying accuracy – positive: compliments, delightful, pleasantly, stars, captivating, excellent, awesome, beautifully, comfy, perfect. 
Revised: • Ten terms annotated as useful with highest underlying accuracy – Class=1: compliments, delightful, pleasantly, stars, captivating, excellent, awesome, beautifully, comfy, perfect. – Class=0: poorly, worthless, disappointing, refund, waste, yuck, garbage, unusable, useless, junk. • Ten terms annotated as useful by users, selected at random – Class=1: interesting, beautifully, value, loves, strong, expected, gorgeous, perfectly, durable, great.

Original: – negative: poorly, worthless, disappointing, refund, waste, yuck, garbage, unusable, useless, junk.
Revised: – Class=0: sent, zero, money, mess, crap, refund, wasted, joke, unusable, beware B.3. IWS WITH AN ORACLE , OCCUPATION CLASSIFICATION

Original: • Ten terms  annotated as useful by users, selected at random – Positive: interesting, beautifully, value, loves, strong, expected, gorgeous, perfectly, durable, great.
Revised: We here provide examples of heuristics found during the IWS-LSE procedure using a simulated oracle, on the BiasBios biographies datasets. We believe that real users (’internet biography experts’) would be able to make very similar distinctions. For the ’Bias Bios: journalist or photographer’ dataset, the ten terms most frequently annotated as useful by the oracle were:

Original: – Negative: sent, zero, money, mess, crap, refund, wasted, joke, unusable, beware.
Revised: • Class = 1: photography, clients, fashion, studio, photographer, art, commercial, ﬁne, creative, advertising.

Original: A. O RACLE EXPERIMENTS ABLATION
Revised: • Class = 0: journalism, writing, reporting, news, media, writer, writes, editor, reporter, newspaper. For the ’Bias Bios: painter or architect’ dataset. Ten terms most frequently annotated as useful by oracle: • Class = 1: commercial, buildings, development, residential, planning, architects, ﬁrm, master, design, construction.

Revised: • Class = 0: painting, museum, collections, exhibition, gallery, born, artists, shows, series, art.

Revised: For the ’Bias Bios: professor or physician’ dataset, the ten terms most frequently annotated as useful by the oracle were:

Revised: • Class = 1: medical, orthopaedic, residency, family, practice, surgery, memorial, general, physician, saint.

Revised: • Class = 0: studies, phd, science, teaching, engineering, received, interests, member, published, professor.

Revised: For the ’Bias Bios: professor or teacher’ dataset, the ten terms most frequently annotated as useful by the oracle were:

Revised: • Class = 1: students, english, teacher, schools, enjoys, years, life, classroom, children, elementary.

Revised: • Class = 0: review, research, interests, published, editor, university, journals, associate, studies, phd.

Original: In this section we provide resultsfor all datasets of ablation experiments we conducted.
Revised: In this section we provide results of ablation experiments for IWS.

Original: The IWS-LSE algorithm requires us to set a thresholdabove random accuracy r  around which our model aims to partition the dataset.
Revised: The IWS-LSE algorithm requires us to set a threshold r on the (unknown) LF accuracy around which our model aims to partition the set of candidate LFs.

Original: Fig.
Revised: Fig.

Original: 7 provides results for different  threshold settings  and  shows that the algorithm performs well across a wide range.
Revised: 8 provides results for different r threshold settings for IWS-LSE-a and IWSLSE-ac, correspondin to Scenario (A) and Scenario (B). The ﬁgure shows that the algorithms perform well across a wide range of r .

Original: While there is no clear, distinct performance difference discernible, the results suggest that ahigh threshold r 0 . {+
+}
Revised: While there is no clear, distinct performance difference discernible, the ﬁgure suggest that a threshold too close to 1 . {+
+}

Original: 8 causes the algorithm to under-perform, potentially as it stiﬂes exploration of LFs.
Revised: can cause the algorithm to under-perform. A possible explanation is that as it stiﬂes exploration of LFs within the limited budget of queries to users.

Original: In Scenario (B), which corresponds to the LSE-ac and where we aim to ﬁnd a ﬁnal set of LFs of limited size, Fig. 8 shows that a wide range (50 to 200) of ﬁnal set sizes produce good results.
Revised: In Scenario (B), which corresponds to the IWS-LSE-ac algorithm, our aim to ﬁnd a ﬁnal set of LFs of limited size. Fig. 9 shows that a wide range ( ˜ m = 50 to 200) of ﬁnal set sizes produce good results.

Revised: Recall that in our experiments, we bound the size of the ﬁnal set of LFs at each iteration t by m = (cid:80) t − 1 i =1 u i + ˜ m , i.e

Revised: [Figure]

Revised: On all datasets test set performance is very similar after around 100 iterations

Original: [Figure]
Revised: [Figure]

Revised: showing that a wide range of such thresholds leads to good test set performance.

Revised: IWS

Original: [Figure]
Revised: [Figure]

Original: B O N L ABELING F UNCTION P ROPERTIES
Revised: C O N L ABELING F UNCTION P ROPERTIES

Original: In this section we analyze how LF accuracy and LF propensity (i.e. non-abstain behavior) inﬂuence the estimate of the true latent class label Y ∗ .
Revised: In this section we analyze how LF accuracy and LF propensity (i.e. non-abstain behavior) inﬂuence the estimate of the true latent class label Y ∗ .

Original: We focus on the binary classiﬁcation case for simplicity.
Revised: We focus on the binary classiﬁcation case for simplicity.

Original: Assume each data point x ∈ X has a latent class label y ∗ ∈ Y
Revised: Assume each data point x ∈ X has a latent class label y ∗ ∈ Y

Original: = {− 1 , 1 } .
Revised: = {− 1 , 1 } .

Original: Given n unlabeled, i.i.d. data points X
Revised: Given n unlabeled, i.i.d. data points X

Original: = { x i } ni =1 , our goal is to train a classiﬁer f : X → Y such that f ( x ) =
Revised: = { x i } ni =1 , our goal is to train a classiﬁer f : X → Y such that f ( x ) =

Original: y ∗ .
Revised: y ∗ .

Original: As in Ratner et al.
Revised: As in Ratner et al.

Original: a user provides m LFs { λ j } mj =1 , where λ j : X → Y ∪ { 0 } noisily label the data with λ j ( x ) ∈ {− 1 , 1 } or abstain with λ j ( x ) = 0 .
Revised: a user provides m LFs { λ j } mj =1 , where λ j : X → Y ∪ { 0 } noisily label the data with λ j ( x ) ∈ {− 1 , 1 } or abstain with λ j ( x ) = 0 .

Original: The corresponding LF output matrix is Λ ∈ {− 1 , 0 , 1 } n × m , where Λ i,j = λ j ( x i ) .
Revised: The corresponding LF output matrix is Λ ∈ {− 1 , 0 , 1 } n × m , where Λ i,j = λ j ( x i ) .

Original: We deﬁne a factor graph as proposed in Ratner et al.
Revised: We deﬁne a factor graph as proposed in Ratner et al.

Original: (2016; 2020) to obtain probabilistic labels by modeling the LF accuracies via factor φ Labi,j (Λ , Y ) (cid:44) 1 { Λ ij (cid:54) = 0 } and labeling propensity by factor φ Acci,j (Λ , Y ) (cid:44) 1 { Λ ij = y i } , and for simplicity assume LFs are independent conditional on Y .
Revised: (2016; 2020) to obtain probabilistic labels by modeling the LF accuracies via factor φ Labi,j (Λ , Y ) (cid:44) 1 { Λ ij (cid:54) = 0 } and labeling propensity by factor φ Acci,j (Λ , Y ) (cid:44) 1 { Λ ij = y i } , and for simplicity assume LFs are independent conditional on Y .

Original: The label model is deﬁned as
Revised: The label model is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: where Z θ is a normalizing constant and φ i (Λ i , y i ) deﬁned to be the concatenation of the factors for all LFs j = 1 , . . .
Revised: where Z θ is a normalizing constant and φ i (Λ i , y i ) deﬁned to be the concatenation of the factors for all LFs j = 1 , . . .

Original: , m for sample i .
Revised: , m for sample i .

Original: Also, let θ =
Revised: Also, let θ =

Original: ( θ (1) , θ (2) )
Revised: ( θ (1) , θ (2) )

Original: where θ (1) , θ (2) ∈ R m .
Revised: where θ (1) , θ (2) ∈ R m .

Original: Here, θ (1) are the canonical parameters for the LF accuracies, and θ (2) the canonical parameters for the LF propensities.
Revised: Here, θ (1) are the canonical parameters for the LF accuracies, and θ (2) the canonical parameters for the LF propensities.

Original: To estimate the label model parameters, we generally obtain the maximum marginal likelihood estimate via the (scaled) log likelihood
Revised: To estimate the label model parameters, we generally obtain the maximum marginal likelihood estimate via the (scaled) log likelihood

Original: [Equation]
Revised: [Equation]

Original: Let ﬁnite ˆ θ ∈ R 2 m be such an estimate.
Revised: Let ﬁnite ˆ θ ∈ R 2 m be such an estimate.

Original: We use p ˆ θ ( y | Λ i ) to obtain probabilistic labels:
Revised: We use p ˆ θ ( y | Λ i ) to obtain probabilistic labels:

Original: [Equation]
Revised: [Equation]

Original: Note that the label estimate does not directly depend on θ (2) .
Revised: Note that the label estimate does not directly depend on θ (2) .

Original: Further, note that the denominator is the same over different label possibilities.
Revised: Further, note that the denominator is the same over different label possibilities.

Original: Finally, note that even in a case where we include correlation factors φ Corri,j,k
Revised: Finally, note that even in a case where we include correlation factors φ Corri,j,k

Original: (Λ , Y )
Revised: (Λ , Y )

Original: = 1 { Λ ij = Λ ik } , ( j, k ) ∈ C in the model above with C as a set of potential dependencies
Revised: = 1 { Λ ij = Λ ik } , ( j, k ) ∈ C in the model above with C as a set of potential dependencies

Original: , the probabilistic label will only directly depend on the estimated canonical accuracy parameters θ (1) .
Revised: , the probabilistic label will only directly depend on the estimated canonical accuracy parameters θ (1) .

Original: In the binary classiﬁcation case, which we assume here, the expression simpliﬁes further.
Revised: In the binary classiﬁcation case, which we assume here, the expression simpliﬁes further.

Original: For k ∈ {− 1 , 1 } :
Revised: For k ∈ {− 1 , 1 } :

Original: [Equation]
Revised: [Equation]

Original: where σ deontes the sigmoid function.
Revised: where σ deontes the sigmoid function.

Original: The probabilistic labels are a softmax in the multi-class classiﬁcation case and, as shown above, simplify to a sigmoid in the binary case.
Revised: The probabilistic labels are a softmax in the multi-class classiﬁcation case and, as shown above, simplify to a sigmoid in the binary case.

Original: An absolute label prediction ˆ y ∈ {− 1 , 1 } is therefore simply a function of
Revised: An absolute label prediction ˆ y ∈ {− 1 , 1 } is therefore simply a function of

Original: [Equation]
Revised: [Equation]

Original: We now introduce some assumptions on the accuracy and error probabilities of labeling functions, similar to the Homogenous Dawid-Skene model (Dawid & Skene, 1979; Li et al., 2013) in crowd sourcing, where label source accuracy is the same across classes and errors are evenly divided with probability mass independent of the true class.
Revised: We now introduce some assumptions on the accuracy and error probabilities of labeling functions, similar to the Homogenous Dawid-Skene model (Dawid & Skene, 1979; Li et al., 2013) in crowd sourcing, where label source accuracy is the same across classes and errors are evenly divided with probability mass independent of the true class.

Original: Under these assumptions, we denote by α j = P ( λ j ( x ) =
Revised: Under these assumptions, we denote by α j = P ( λ j ( x ) =

Original: y ∗ | λ j ( x ) (cid:54) = 0)
Revised: y ∗ | λ j ( x ) (cid:54) = 0)

Original: the accuracy of LF j .
Revised: the accuracy of LF j .

Original: Further, we denote by l j = P ( λ j ( x ) (cid:54) = 0) the labeling propensity of j , i.e. how frequently LF j does not abstain.
Revised: Further, we denote by l j = P ( λ j ( x ) (cid:54) = 0) the labeling propensity of j , i.e. how frequently LF j does not abstain.

Original: The observed LF propensity is also referred to as LF coverage in the related literature.
Revised: The observed LF propensity is also referred to as LF coverage in the related literature.

Original: We recall theorem 3.1:
Revised: We recall theorem 3.1:

Original: Theorem 3.1.
Revised: Theorem 3.1.

Original: Assume a binary classiﬁcation setting, m independent labeling functions with accuracy α j ∈ [0 , 1] and labeling propensity l j ∈ [0 , 1] .
Revised: Assume a binary classiﬁcation setting, m independent labeling functions with accuracy α j ∈ [0 , 1] and labeling propensity l j ∈ [0 , 1] .

Original: For a label model as in Eq.
Revised: For a label model as in Eq.

Original: (1) with given label model parameters ˆ θ ∈ R 2 m , and for any i ∈ { 1 , . . . , n } ,
Revised: (1) with given label model parameters ˆ θ ∈ R 2 m , and for any i ∈ { 1 , . . . , n } ,

Original: [Equation]
Revised: [Equation]

Original: where ˆ θ (1) are the m weights of φ Acc , and ˆ y i ∈ {− 1 , 1 } is the label model estimate for y ∗ i .
Revised: where ˆ θ (1) are the m weights of φ Acc , and ˆ y i ∈ {− 1 , 1 } is the label model estimate for y ∗ i .

Original: Proof.
Revised: Proof.

Original: Assume that we use the label model to obtain a label estimate ˆ y i ∈ {− 1 , 1 } .
Revised: Assume that we use the label model to obtain a label estimate ˆ y i ∈ {− 1 , 1 } .

Original: As shown in
Revised: As shown in

Original: Eq.
Revised: Eq.

Original: (10), the prediction rule in that case is
Revised: (10), the prediction rule in that case is

Original: [Equation]
Revised: [Equation]

Original: Deﬁne by
Revised: Deﬁne by

Original: [Equation]
Revised: [Equation]

Original: m ( x ))
Revised: m ( x ))

Original: the vector of the j = 1 , . . .
Revised: the vector of the j = 1 , . . .

Original: , m LF outputs on x .
Revised: , m LF outputs on x .

Original: Further, we deﬁne for k ∈ {− 1 , 1 } :
Revised: Further, we deﬁne for k ∈ {− 1 , 1 } :

Original: [Equation]
Revised: [Equation]

Original: For the two label options k ∈ {− 1 , 1 } , we have
Revised: For the two label options k ∈ {− 1 , 1 } , we have

Original: [Equation]
Revised: [Equation]

Original: ˆ
Revised: ˆ

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: V ˆ θ ( λ ( x ) , −
Revised: V ˆ θ ( λ ( x ) , −

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: Now, we want to obtain a bound on the probability that the label estimate ˆ y i is equal to the true label.
Revised: Now, we want to obtain a bound on the probability that the label estimate ˆ y i is equal to the true label.

Original: We have
Revised: We have

Original: P
Revised: P

Original: [Equation]
Revised: [Equation]

Original: = P
Revised: = P

Original: [Equation]
Revised: [Equation]

Original: |
Revised: |

Original: [Equation]
Revised: [Equation]

Original: 1 | y ∗ i = − 1) .
Revised: 1 | y ∗ i = − 1) .

Original: Note that
Revised: Note that

Original: P
Revised: P

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: and that
Revised: and that

Original: [Equation]
Revised: [Equation]

Original: We therefore have
Revised: We therefore have

Original: [Equation]
Revised: [Equation]

Original: Now we deﬁne ξ ij =
Revised: Now we deﬁne ξ ij =

Original: ˆ θ (1) j λ j ( x i )
Revised: ˆ θ (1) j λ j ( x i )

Original: and we know that ξ j ∈ [ −| ˆ θ (1) j | , | ˆ θ (1) j | ] .
Revised: and we know that ξ j ∈ [ −| ˆ θ (1) j | , | ˆ θ (1) j | ] .

Original: Given the Dawid-Skene model assumptions stated previously, we have
Revised: Given the Dawid-Skene model assumptions stated previously, we have

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Now, using Hoeffding’s inequality and assuming independent labeling functions, we can bound
Revised: Now, using Hoeffding’s inequality and assuming independent labeling functions, we can bound

Original: P (ˆ y i = 1 | y ∗ i = 1) and P (ˆ y i = − 1 | y ∗ i = − 1) from below:
Revised: P (ˆ y i = 1 | y ∗ i = 1) and P (ˆ y i = − 1 | y ∗ i = − 1) from below:

Original: [Equation]
Revised: [Equation]

Original: λ j ( x i ) > 0 | y ∗ i = 1) =
Revised: λ j ( x i ) > 0 | y ∗ i = 1) =

Original: P ( m (cid:88)
Revised: P ( m (cid:88)

Original: j = ξ ij > 0 | y ∗
Revised: j = ξ ij > 0 | y ∗

Original: i = 1) =
Revised: i = 1) =

Original: P ( m (cid:88)
Revised: P ( m (cid:88)

Original: j = ξ ij − E [ m (cid:88)
Revised: j = ξ ij − E [ m (cid:88)

Original: j = ξ ij | y ∗ i = 1] > − m (cid:88)
Revised: j = ξ ij | y ∗ i = 1] > − m (cid:88)

Original: j
Revised: j

Original: = ˆ θ (1) j l j (2 ∗ α j − 1) | y ∗ i = 1) ≥ 1 − exp (cid:32) − ( (cid:80) mj
Revised: = ˆ θ (1) j l j (2 ∗ α j − 1) | y ∗ i = 1) ≥ 1 − exp (cid:32) − ( (cid:80) mj

Original: =1 ˆ θ (1) j (2 α j − 1) l j )
Revised: =1 ˆ θ (1) j (2 α j − 1) l j )

Original: 2 || ˆ θ (1) ||(cid:33) , and
Revised: 2 || ˆ θ (1) ||(cid:33) , and

Original: P ( m (cid:88)
Revised: P ( m (cid:88)

Original: j
Revised: j

Original: = ˆ θ (1) j λ j ( x i ) < 

Revised: = ˆ θ (1) j λ j ( x i ) <

Original: | y ∗ i = − 1)
Revised: | y ∗ i = − 1)

Original: =
Revised: =

Original: P ( m (cid:88)
Revised: P ( m (cid:88)

Original: j = ξ ij < 0 | y ∗ i = − 1) =
Revised: j = ξ ij < 0 | y ∗ i = − 1) =

Original: P ( m (cid:88)
Revised: P ( m (cid:88)

Original: j = ξ ij − E [ m (cid:88)
Revised: j = ξ ij − E [ m (cid:88)

Original: j = ξ ij | y ∗ i = − 1] < m (cid:88)
Revised: j = ξ ij | y ∗ i = − 1] < m (cid:88)

Original: j
Revised: j

Original: = ˆ θ (1) j l j (2 ∗ α j − 1) | y ∗ i = − 1) ≥ 1 − exp (cid:32) − ( (cid:80) mj
Revised: = ˆ θ (1) j l j (2 ∗ α j − 1) | y ∗ i = − 1) ≥ 1 − exp (cid:32) − ( (cid:80) mj

Original: =1 ˆ θ (1) j (2 α j − 1) l j )
Revised: =1 ˆ θ (1) j (2 α j − 1) l j )

Original: 2 || ˆ θ (1) ||(cid:33) .
Revised: 2 || ˆ θ (1) ||(cid:33) .

Original: Finally we have
Revised: Finally we have

Original: P (
Revised: P (

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: ) ||(cid:33) .
Revised: ) ||(cid:33) .

Original: What do the theorem and the quantities analyzed in this section indicate?
Revised: What do the theorem and the quantities analyzed in this section indicate?

Original: • The trade-off between LF accuracy and LF propensity (also referred to as LF coverage) is captured by (2 α j − 1) l j which allows us to rank LFs if we know the accuracy α j or can estimate it and use the observed, empirical coverage as an estimate of l j .
Revised: • The trade-off between LF accuracy and LF propensity (also referred to as LF coverage) is captured by (2 α j − 1) l j which allows us to rank LFs if we know the accuracy α j or can estimate it and use the observed, empirical coverage as an estimate of l j .

Original: • Not surprising, the relation between sign ( θ j ) and α j is important.
Revised: • Not surprising, the relation between sign ( θ j ) and α j is important.

Original: A better than random LF j should have a positive θ j .
Revised: A better than random LF j should have a positive θ j .

Original: This indicates that a gap to randomness is important if we cannot guarantee that we learn θ j well, to reduce the chance of obtaining a negative θ j for better than random LF j , or vice versa.
Revised: This indicates that a gap to randomness is important if we cannot guarantee that we learn θ j well, to reduce the chance of obtaining a negative θ j for better than random LF j , or vice versa.

Original: • Note how the label estimates are obtained in Eq.
Revised: • Note how the label estimates are obtained in Eq.

Original: Increasing the θ j of am LF also effectively means reducing the impact other LFs have on a prediction.
Revised: Increasing the θ j of am LF also effectively means reducing the impact other LFs have on a prediction.

Original: In particular when θ estimates are imperfect, a gap to random accuracy of α j is important to obtain good label estimates.
Revised: In particular when θ estimates are imperfect, a gap to random accuracy of α j is important to obtain good label estimates.

Original: Intuitively, we do not want to add excessive noise by including LFs close to random unless we can guarantee that their parameter estimate is appropriately low and has the correct sign.
Revised: Intuitively, we do not want to add excessive noise by including LFs close to random unless we can guarantee that their parameter estimate is appropriately low and has the correct sign.

Original: B. W HAT ARE THE OPTIMAL θ PARAMETERS ?
Revised: C. W HAT ARE THE OPTIMAL LABEL MODEL PARAMETERS ?

Original: Here we discuss the optimal θ parameters for two cases: ﬁrst for when we assume that the label model factor graph consists only of accuracy factors and second for when this label model also takes
Revised: Here we discuss the optimal θ parameters for two cases: ﬁrst for when we assume that the label model factor graph consists only of accuracy factors and second for when this label model also takes

Original: LF propensity into account.
Revised: LF propensity into account

Original: B.1. A SSUMING ACCURACY FACTORS ONLY
Revised: . C.1. A SSUMING ACCURACY FACTORS ONLY

Original: In the previous section we assumed that we are given estimated θ parameters.
Revised: In the previous section we assumed that we are given estimated θ parameters.

Original: Naturally, we may next ask ourselves what the optimal theta parameters are.
Revised: Naturally, we may next ask ourselves what the optimal theta parameters are.

Original: Let us start with a simple case.
Revised: Let us start with a simple case.

Original: Let Y , H be random variables of the class variable and vector of LFs, respectively.
Revised: Let Y , H be random variables of the class variable and vector of LFs, respectively.

Original: Assume that we only model
Revised: Assume that we only model

Original: LF accuracy and deﬁne 1 { λ ( x )=
Revised: LF accuracy and deﬁne 1 { λ ( x )=

Original: y } to be the element-wise indicator function.
Revised: y } to be the element-wise indicator function.

Original: Also, assume that the true distribution can be expressed by the following model: 
Revised: Also, assume that the true distribution can be expressed by the following model: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Note that Z can be written as
Revised: Note that Z can be written as

Original: [Equation]
Revised: [Equation]

Original: where L is the number of classes.
Revised: where L is the number of classes.

Original: Now, note that
Revised: Now, note that

Original: [Equation]
Revised: [Equation]

Original: Thus
Revised: Thus

Original: [Equation]
Revised: [Equation]

Original: P ( H j = Y ) =
Revised: P ( H j = Y ) =

Original: e θj e θ j + L .
Revised: e θj e θ j + L .

Original: Which implies
Revised: Which implies

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Further [Equation] note that P 
Revised: Further, note that P ( H

Original: [Equation]
Revised: [Equation]

Original:  = 0)
Revised: (cid:54) = 0)

Original: =
Revised: =

Original: P ( H j = Y
Revised: P ( H j = Y

Original: | H j (cid:54) = 0) P ( H j (cid:54) = 0) is a combination of an LF’s accuracy and label propensity.
Revised: | H j (cid:54) = 0) P ( H j (cid:54) = 0) is a combination of an LF’s accuracy and label propensity.

Original: B.1. A SSUMING ACCURACY FACTORS AND LF PROPENSITY FACTORS
Revised: C.1. A SSUMING ACCURACY FACTORS AND LF PROPENSITY FACTORS

Original: We now analyze a slightly more elaborate label model which also models LF propensity.
Revised: We now analyze a slightly more elaborate label model which also models LF propensity.

Original: Again, Let
Revised: Again, Let

Original: Y, H denote the random variables for class labels and LFs, respectively
Revised: Y, H denote the random variables for class labels and LFs, respectively

Original: [Equation]
Revised: [Equation]

Original: Z can be written as
Revised: Z can be written as

Original: [Equation]
Revised: [Equation]

Original: The likelihood of a correct vote is given by
Revised: The likelihood of a correct vote is given by

Original: [Equation]
Revised: [Equation]

Original: Further note that
Revised: Further note that

Original: [Equation]
Revised: [Equation]

Original: So we can express
Revised: So we can express

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: Solving for θ (1) j and θ (2) j , we ﬁnd θ (1) ∗ j = ln (cid:18)
Revised: Solving for θ (1) j and θ (2) j , we ﬁnd θ (1) ∗ j = ln (cid:18)

Original: ( L − 1) α j 1 − α j (cid:19) (28) θ (2) ∗ j = ln (cid:18) (1 − α j ) l j ( L − 1)(1 − l j ) (cid:19) , (29) where α j = P ( H j = Y | H j (cid:54)
Revised: ( L − 1) α j 1 − α j (cid:19) (28) θ (2) ∗ j = ln (cid:18) (1 − α j ) l j ( L − 1)(1 − l j ) (cid:19) , (29) where α j = P ( H j = Y | H j (cid:54)

Original: = 0)
Revised: = 0)

Original: and l j = P ( H j (cid:54) = 0) .
Revised: and l j = P ( H j (cid:54) = 0) .

Original: Note that in the binary case θ (1) ∗ j is positive only when α j > 0 .
Revised: Note that in the binary case θ (1) ∗ j is positive only when α j > 0 .