Original: ReGAL: Rule-Generative Active Learning for Model-in-the-Loop Weak Supervision
Revised: ReGAL: Rule-Generative Active Learning for Model-in-the-Loop Weak Supervision

Original: Abstract
Revised: Abstract

Original: One of the main bottlenecks to extending deep learning systems to new domains is the prohibitive cost of acquiring sufﬁcient training labels.
Revised: One of the main bottlenecks to extending deep learning systems to new domains is the prohibitive cost of acquiring sufﬁcient training labels.

Original: While many previous works have sought to alleviate this problem with weak supervision and data programming, rule and label noise prevent them from approaching fully-supervised performance.
Revised: While many previous works have sought to alleviate this problem with weak supervision and data programming, rule and label noise prevent them from approaching fully-supervised performance.

Original: This work-in-progress provides a principled, AI-guided approach to improve rule-based and weakly supervised text classiﬁcation by performing active learning not on individual data instances, but on entire labeling functions.
Revised: This work-in-progress provides a principled, AI-guided approach to improve rule-based and weakly supervised text classiﬁcation by performing active learning not on individual data instances, but on entire labeling functions.

Original: We argue that such a framework can guide users and subject matter experts to select labeling rules that expand label function coverage without sacriﬁcing clarity.
Revised: We argue that such a framework can guide users and subject matter experts to select labeling rules that expand label function coverage without sacriﬁcing clarity.

Original: Our experiments show that our framework, ReGAL, is able to generate coherent labeling rules while simultaneously obtaining state-of-the-art performance in weakly supervised text classiﬁcation.
Revised: Our experiments show that our framework, ReGAL, is able to generate coherent labeling rules while simultaneously obtaining state-of-the-art performance in weakly supervised text classiﬁcation.

Original:  Introduction
Revised:  Introduction

Original: One of the primary criticisms that has been leveled against deep learning is the high volume of data needed to produce models that generalize well and avoid overﬁtting.
Revised: One of the primary criticisms that has been leveled against deep learning is the high volume of data needed to produce models that generalize well and avoid overﬁtting.

Original: Numerous approaches have been proposed to solve this problem, including transfer learning [1], active learning [2, 3], bootstrapping [4], distant supervision [5], rule-based supervision [6, 7] and crowd-sourced labeling [8].
Revised: Numerous approaches have been proposed to solve this problem, including transfer learning [1], active learning [2, 3], bootstrapping [4], distant supervision [5], rule-based supervision [6, 7] and crowd-sourced labeling [8].

Original: While this task is present across all domains of natural language processing (NLP), it is signiﬁcantly more difﬁcult in the highly technical and low resource domains where labels must be curated by subject matter experts (SMEs).
Revised: While this task is present across all domains of natural language processing (NLP), it is signiﬁcantly more difﬁcult in the highly technical and low resource domains where labels must be curated by subject matter experts (SMEs).

Original: Low-resource languages often have a very small pool of potential annotators while technical domains such as scholarly articles contain such a wide breadth of information in literature means that a single SME cannot annotate a single corpus by himself or herself.
Revised: Low-resource languages often have a very small pool of potential annotators while technical domains such as scholarly articles contain such a wide breadth of information in literature means that a single SME cannot annotate a single corpus by himself or herself.

Original: These factors, combined with the high cost of SME expertise motivates the development of systems that maximize both the speed and information content of annotation, thus allowing annotators to impart more valuable data in shorter time.
Revised: These factors, combined with the high cost of SME expertise motivates the development of systems that maximize both the speed and information content of annotation, thus allowing annotators to impart more valuable data in shorter time.

Original: In this work, we develop a model called ReGAL for Rule-Generative Active Learning which accelerates data labeling by interactively soliciting human feedback on labeling functions instead of individual data points.
Revised: In this work, we develop a model called ReGAL for Rule-Generative Active Learning which accelerates data labeling by interactively soliciting human feedback on labeling functions instead of individual data points.

Original: This framework, depicted in Figure 1 allows our active learner to seek feedback on model weaknesses while simultaneously labeling large swaths of examples.
Revised: This framework, depicted in Figure 1 allows our active learner to seek feedback on model weaknesses while simultaneously labeling large swaths of examples.

Original:  Preliminaries 2.1 Problem Formulation
Revised:  Preliminaries 2.1 Problem Formulation

Original: We assume that we are given a set of documents D
Revised: We assume that we are given a set of documents D

Original: = { d 1 , d 2 , . . .
Revised: = { d 1 , d 2 , . . .

Original: , d | D | } , each of which has a (possibly unknown) classiﬁcation label c i ∈ C .
Revised: , d | D | } , each of which has a (possibly unknown) classiﬁcation label c i ∈ C .

Original: Each document d i =
Revised: Each document d i =

Original: [ v i, 1 , v i, 2 , . .
Revised: [ v i, 1 , v i, 2 , . .

Original: . , v i,T ] represents a sequence of tokens from the vocabulary V , where tokens drawn from V could be words, subwords, characters, etc.
Revised: . , v i,T ] represents a sequence of tokens from the vocabulary V , where tokens drawn from V could be words, subwords, characters, etc.

Original: We assume that we do not have access to ground-truth labels for the documents in the training set but that we have a small number of heuristic labeling functions (LFs; i.e. labeling rules)
Revised: We assume that we do not have access to ground-truth labels for the documents in the training set but that we have a small number of heuristic labeling functions (LFs; i.e. labeling rules)

Original: R = { r 1 , r 2 , . .
Revised: R = { r 1 , r 2 , . .

Original: , r l } , where each r j : D → C ∪ { c abstain } is a function that maps documents to a class label in C or abstains from labeling.
Revised: , r l } , where each r j : D → C ∪ { c abstain } is a function that maps documents to a class label in C or abstains from labeling.

Original: This set of LFs induces a vector of noisy labels for each document, denoted (cid:96) i =
Revised: This set of LFs induces a vector of noisy labels for each document, denoted (cid:96) i =

Original: [ r 1 ( d i ) , r 2 ( d i ) , . . .
Revised: [ r 1 ( d i ) , r 2 ( d i ) , . . .

Original: , r l ( d i )] T .
Revised: , r l ( d i )] T .

Original: Because LFs act as rule-based labelers, we freely interchange the terms “labeling function” and “rule” throughout the paper.
Revised: Because LFs act as rule-based labelers, we freely interchange the terms “labeling function” and “rule” throughout the paper.

Original: 2.2 Challenges
Revised: 2.2 Challenges

Original: Weakly supervised text classiﬁcation presents three main challenges: label noise and label incompleteness , and annotator effort .
Revised: Weakly supervised text classiﬁcation presents three main challenges: label noise and label incompleteness , and annotator effort .

Original: Label Noise Label noise is the problem of labeling functions generating incorrect labels for particular data instances.
Revised: Label Noise Label noise is the problem of labeling functions generating incorrect labels for particular data instances.

Original: This problem generally occurs when a speciﬁed labeling function is too general and thus mislabels into the wrong class.
Revised: This problem generally occurs when a speciﬁed labeling function is too general and thus mislabels into the wrong class.

Original: The diversity of language presents an extremely large space of possible misapplications for a single labeling function, and enumerating them can be prohibitively expensive.
Revised: The diversity of language presents an extremely large space of possible misapplications for a single labeling function, and enumerating them can be prohibitively expensive.

Original: A number of recent works have sought to address this problem via numerous means, including generative label denoising [6, 9], self-training on synthetic examples generated with latent variable models [10], using a neural network to identify improper applications of labeling functions using labeled rule exemplars [11], and active learning on instances with conﬂicting labels [12].
Revised: A number of recent works have sought to address this problem via numerous means, including generative label denoising [6, 9], self-training on synthetic examples generated with latent variable models [10], using a neural network to identify improper applications of labeling functions using labeled rule exemplars [11], and active learning on instances with conﬂicting labels [12].

Original: While some of these methods have shown promising results, there is a distinct need for better methods of automatically solving labeling function conﬂicts.
Revised: While some of these methods have shown promising results, there is a distinct need for better methods of automatically solving labeling function conﬂicts.

Original: Our proposed methodology seeks to reduce label noise by automatically learning rules designed to differentiate between separate classes.
Revised: Our proposed methodology seeks to reduce label noise by automatically learning rules designed to differentiate between separate classes.

Original: Label Incompleteness A complementary problem to label noise is label incompleteness.
Revised: Label Incompleteness A complementary problem to label noise is label incompleteness.

Original: Label incompleteness is the insufﬁciency of labeling functions to assign labels to particular slices of a dataset.
Revised: Label incompleteness is the insufﬁciency of labeling functions to assign labels to particular slices of a dataset.

Original: This occurs when the syntactic and semantic patterns in a subset of examples do not lie within the scope of the given labeling functions.
Revised: This occurs when the syntactic and semantic patterns in a subset of examples do not lie within the scope of the given labeling functions.

Original: Label incompleteness is particularly pervasive in the long tails of a dataset, which may contain more informative data for better model performance and generalization.
Revised: Label incompleteness is particularly pervasive in the long tails of a dataset, which may contain more informative data for better model performance and generalization.

Original: This can be especially manifested in low-resource or highly technical domains where differences in nomenclature could lead to large labeling gaps.
Revised: This can be especially manifested in low-resource or highly technical domains where differences in nomenclature could lead to large labeling gaps.

Original: Approaches to tackle this problem include differentiable soft-matching of labeling rules to unlabeled instances [13], automatic rule generation using pre-speciﬁed rule patterns [14, 15], cotraining a rule-based labeling module with a deep learning module capable of matching unlabeled instances [16, 10], and encouraging LF diversity by interactively soliciting LFs for unlabeled instances [12].
Revised: Approaches to tackle this problem include differentiable soft-matching of labeling rules to unlabeled instances [13], automatic rule generation using pre-speciﬁed rule patterns [14, 15], cotraining a rule-based labeling module with a deep learning module capable of matching unlabeled instances [16, 10], and encouraging LF diversity by interactively soliciting LFs for unlabeled instances [12].

Original: Annotator Effort Many domains such as biomedical or legal NLP require substantial subject matter expertise to annotate correctly.
Revised: Annotator Effort Many domains such as biomedical or legal NLP require substantial subject matter expertise to annotate correctly.

Original: However, SMEs are costly and have limited time to for annotation.
Revised: However, SMEs are costly and have limited time to for annotation.

Original: This problem is strikingly evident when one compares the size of labeled datasets in biomedicine [17, 18, 1] with those for general domain tasks.
Revised: This problem is strikingly evident when one compares the size of labeled datasets in biomedicine [17, 18, 1] with those for general domain tasks.

Original: For example, the average training set in the general domain GLUE benchmark [19] is over 25x larger than that of the average dataset in the contemporaneous BLUE benchmark [1] for biomedical NLP.
Revised: For example, the average training set in the general domain GLUE benchmark [19] is over 25x larger than that of the average dataset in the contemporaneous BLUE benchmark [1] for biomedical NLP.

Original: Accordingly, it is important to solicit the most useful possible SME feedback to maximize label signal while minimizing annotator time and effort.
Revised: Accordingly, it is important to solicit the most useful possible SME feedback to maximize label signal while minimizing annotator time and effort.

Original: By presenting annotators with candidate labeling rules, ReGAL reduces the time necessary to specify rules by hand and thus offers large potential gains to annotator efﬁciency.
Revised: By presenting annotators with candidate labeling rules, ReGAL reduces the time necessary to specify rules by hand and thus offers large potential gains to annotator efﬁciency.

Original: 2.3 Objectives
Revised: 2.3 Objectives

Original: This paper develops a model, ReGAL , that addresses these challenges by automatically proposing labeling rules designed to (1) disambiguate instances with conﬂicting LF-induced labels and (2) extend coverage to unlabeled portions of the dataset.
Revised: This paper develops a model, ReGAL , that addresses these challenges by automatically proposing labeling rules designed to (1) disambiguate instances with conﬂicting LF-induced labels and (2) extend coverage to unlabeled portions of the dataset.

Original: This is done interactively as annotators generate labels, allowing ReGAL to adapt to new labeling needs as the set of labels expands.
Revised: This is done interactively as annotators generate labels, allowing ReGAL to adapt to new labeling needs as the set of labels expands.

Original:  Related Work 3.1 Data programming
Revised:  Related Work 3.1 Data programming

Original: Our work is closely related with recent works surrounding data programming and rule-based text classiﬁcation.
Revised: Our work is closely related with recent works surrounding data programming and rule-based text classiﬁcation.

Original: One of the seminal works in this category proposed Snorkel [20, 6], which introduced a rule-based labeling model where multiple, overlapping rules are proposed for each class and then are denoised with a generative model to account for the inter-rule correlations.
Revised: One of the seminal works in this category proposed Snorkel [20, 6], which introduced a rule-based labeling model where multiple, overlapping rules are proposed for each class and then are denoised with a generative model to account for the inter-rule correlations.

Original: A number of alternative denoising approaches have since been proposed to improve the quality of aggregated labels.
Revised: A number of alternative denoising approaches have since been proposed to improve the quality of aggregated labels.

Original: Among these, Ren et al.
Revised: Among these, Ren et al.

Original: [16] propose a self-training discriminative denoising module, while Awasthi et al.
Revised: [16] propose a self-training discriminative denoising module, while Awasthi et al.

Original: [11] propose learning appropriate bounds on rule coverage from a small labeled set of data.
Revised: [11] propose learning appropriate bounds on rule coverage from a small labeled set of data.

Original: ReGAL differs from this work by providing model-based assistance to deﬁne high-quality rules and augment human knowledge with global data patterns in addition to providing a module to denoise labeling rules.
Revised: ReGAL differs from this work by providing model-based assistance to deﬁne high-quality rules and augment human knowledge with global data patterns in addition to providing a module to denoise labeling rules.

Original: Follow-up studies [12] on Snorkel have also shown that Snorkel users tended to select rules by looking at individual labeled instances and seeking to create rules reﬂecting the patterns they observe in the data.
Revised: Follow-up studies [12] on Snorkel have also shown that Snorkel users tended to select rules by looking at individual labeled instances and seeking to create rules reﬂecting the patterns they observe in the data.

Original: This study further demonstrated that label efﬁciency can be improved by showing users currently unlabeled instances and instances with high label conﬂict, i.e. seeking labels in highly valuable areas of the training data.
Revised: This study further demonstrated that label efﬁciency can be improved by showing users currently unlabeled instances and instances with high label conﬂict, i.e. seeking labels in highly valuable areas of the training data.

Original: ReGAL provides the possibility to extend this by automatically proposing rules suited to conﬂicted or unlabeled data slices, thus reducing the manual effort involved.
Revised: ReGAL provides the possibility to extend this by automatically proposing rules suited to conﬂicted or unlabeled data slices, thus reducing the manual effort involved.

Original: 3.2 Rule Learning
Revised: 3.2 Rule Learning

Original: Another key area of related work is rule learning, in which models seek to automatically label datasets by generating their own label rules.
Revised: Another key area of related work is rule learning, in which models seek to automatically label datasets by generating their own label rules.

Original: MetaPad [21] mined and consolidated information extraction patterns from large text corpora to improve the rule-based extraction of speciﬁc types of named entities and their attributes from raw text.
Revised: MetaPad [21] mined and consolidated information extraction patterns from large text corpora to improve the rule-based extraction of speciﬁc types of named entities and their attributes from raw text.

Original: REPEL [15], on the other hand, focused speciﬁcally on rule-guided relation extraction.
Revised: REPEL [15], on the other hand, focused speciﬁcally on rule-guided relation extraction.

Original: The authors began with small sets of labeled seed instances and iteratively reﬁned a set of potential candidate rules based on keywords in the shortest dependency path between entities.
Revised: The authors began with small sets of labeled seed instances and iteratively reﬁned a set of potential candidate rules based on keywords in the shortest dependency path between entities.

Original: While their approach has many similarities to our own, it is limited in that it can only produce a single type of rule and does not allow for iterative reﬁnement via human guidance.
Revised: While their approach has many similarities to our own, it is limited in that it can only produce a single type of rule and does not allow for iterative reﬁnement via human guidance.

Original: Snuba [14] generates candidate rules using a collection of weak learner primitives (e.g. decision stumps, k-nearest neighbors) and then synthesizes and prunes this set of rules to generate ﬁnal rules for labeling.
Revised: Snuba [14] generates candidate rules using a collection of weak learner primitives (e.g. decision stumps, k-nearest neighbors) and then synthesizes and prunes this set of rules to generate ﬁnal rules for labeling.

Original: ReGAL combines elements of this framework to allow iterative communication between downstream classiﬁers and rule selectors to allow each to mutually enhance the other.
Revised: ReGAL combines elements of this framework to allow iterative communication between downstream classiﬁers and rule selectors to allow each to mutually enhance the other.

Original: ReGAL differs from these by soliciting user feedback on rules, thus enabling it complement model-generated LFs with annotator guidance.
Revised: ReGAL differs from these by soliciting user feedback on rules, thus enabling it complement model-generated LFs with annotator guidance.

Original:  Methods
Revised:  Methods

Original: Here we describe ReGAL , a model that interactively generates labeling functions via repeated rounds of user feedback.
Revised: Here we describe ReGAL , a model that interactively generates labeling functions via repeated rounds of user feedback.

Original: This model is composed of three modules: (1) A contextualized word and document encoder that produces global document embeddings and contextualized word embeddings; (2) a rule denoising module that aggregates labeling functions to produce coherent labels for matched and unmatched documents; and (3) a rule proposal module that generates candidate labeling functions.
Revised: This model is composed of three modules: (1) A contextualized word and document encoder that produces global document embeddings and contextualized word embeddings; (2) a rule denoising module that aggregates labeling functions to produce coherent labels for matched and unmatched documents; and (3) a rule proposal module that generates candidate labeling functions.

Original: A depiction of our model is shown in Figure
Revised: A depiction of our model is shown in Figure

Original: 4.1 Contextualized Token Encoder
Revised: 4.1 Contextualized Token Encoder

Original: ReGAL begins with a token encoder.
Revised: ReGAL begins with a token encoder.

Original: The purpose of the encoder is to produce expressive, contextualized token embeddings containing the semantic content necessary to identify potential keywords in generated LFs.
Revised: The purpose of the encoder is to produce expressive, contextualized token embeddings containing the semantic content necessary to identify potential keywords in generated LFs.

Original: These embeddings will also be aggregated by the rule denoiser into a cohesive document representation.
Revised: These embeddings will also be aggregated by the rule denoiser into a cohesive document representation.

Original: We use transformers [22] to construct our encoder because they allow each token to draw contextual information from every other token in its same input document.
Revised: We use transformers [22] to construct our encoder because they allow each token to draw contextual information from every other token in its same input document.

Original: Speciﬁcally, we use a pretrained
Revised: Speciﬁcally, we use a pretrained

Original: BERT-base [23] model as our encoder.
Revised: BERT-base [23] model as our encoder.

Original: We use the outputs
Revised: We use the outputs

Original: h i,t of the last layer as token embeddings: (cid:104) h
Revised: h i,t of the last layer as token embeddings: (cid:104) h

Original: i, 1 , . . .
Revised: i, 1 , . . .

Original: , h i,T (cid:105) =
Revised: , h i,T (cid:105) =

Original: enc ([ v i, 1 , . . .
Revised: enc ([ v i, 1 , . . .

Original: , v i,T ])
Revised: , v i,T ])

Original: We will henceforth let H i =
Revised: We will henceforth let H i =

Original: (cid:104)
Revised: (cid:104)

Original: h
Revised: h

Original: i, 1 , . . .
Revised: i, 1 , . . .

Original: , h i,T (cid:105) represent the sequence token embeddings from document d i .
Revised: , h i,T (cid:105) represent the sequence token embeddings from document d i .

Original: In addition to initializing this module with a BERT-base, we encourage the encoder to learn contextual information about rules.
Revised: In addition to initializing this module with a BERT-base, we encourage the encoder to learn contextual information about rules.

Original: We do so using masked language modeling (MLM), where our masking budget consists of all tokens used in keyword LFs as well as a random 10% of tokens from the sequence.
Revised: We do so using masked language modeling (MLM), where our masking budget consists of all tokens used in keyword LFs as well as a random 10% of tokens from the sequence.

Original: We either mask or noise each token according to the strategy in [23] and require our model to predict the correct token in each case.
Revised: We either mask or noise each token according to the strategy in [23] and require our model to predict the correct token in each case.

Original: We optimize this using cross entropy loss over the masked/noised tokens, which we denote L MLM .
Revised: We optimize this using cross entropy loss over the masked/noised tokens, which we denote L MLM .

Original: 4.2 Rule Denoiser
Revised: 4.2 Rule Denoiser

Original: We develop a rule denoiser module similar to [16] that allows our model to learn document-level applicability of matched LFs.
Revised: We develop a rule denoiser module similar to [16] that allows our model to learn document-level applicability of matched LFs.

Original: This is co-trained with a neural prediction module that extends coverage to unmatched documents based on their learned semantic properties.
Revised: This is co-trained with a neural prediction module that extends coverage to unmatched documents based on their learned semantic properties.

Original: The rule denoiser also learns class-speciﬁc token importance scores that will be used to generate new LFs.
Revised: The rule denoiser also learns class-speciﬁc token importance scores that will be used to generate new LFs.

Original: The rule denoiser has two submodules which we discuss in detail: a token attention module and a rule attention module.
Revised: The rule denoiser has two submodules which we discuss in detail: a token attention module and a rule attention module.

Original: 4.2.1 Token attention
Revised: 4.2.1 Token attention

Original: The purpose of the TokenAttention submodule is to learn the token and document level information necessary to generate expressive, class-speciﬁc labeling functions.
Revised: The purpose of the TokenAttention submodule is to learn the token and document level information necessary to generate expressive, class-speciﬁc labeling functions.

Original: Speciﬁcally, TokenAttention computes class-speciﬁc attention scores on each token, and the document’s probability of belonging to each class.
Revised: Speciﬁcally, TokenAttention computes class-speciﬁc attention scores on each token, and the document’s probability of belonging to each class.

Original: These class probabilities also serve as suitability scores of how well-equipped a document is to generate LF keywords for a given class.
Revised: These class probabilities also serve as suitability scores of how well-equipped a document is to generate LF keywords for a given class.

Original: TokenAttention takes as inputs the token embeddings from the document encoder and produces class-speciﬁc token attention a ( c ) i,t , document embeddings e i , and document-level class probabilities p
Revised: TokenAttention takes as inputs the token embeddings from the document encoder and produces class-speciﬁc token attention a ( c ) i,t , document embeddings e i , and document-level class probabilities p

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: (cid:105) .
Revised: (cid:105) .

Original: We describe how each of these is computed as follows.
Revised: We describe how each of these is computed as follows.

Original: We begin by calculating class-speciﬁc attention scores for each token in our document.
Revised: We begin by calculating class-speciﬁc attention scores for each token in our document.

Original: These will be used by our rule proposal network to generate new labeling rules.
Revised: These will be used by our rule proposal network to generate new labeling rules.

Original: They are calculated as
Revised: They are calculated as

Original: [Equation]
Revised: [Equation]

Original: where W a 1 ∈ R m 2 × m 1 and W a 2 ∈ R c × m 2 .
Revised: where W a 1 ∈ R m 2 × m 1 and W a 2 ∈ R c × m 2 .

Original: These scores are then used to calculate a class-speciﬁc document representation
Revised: These scores are then used to calculate a class-speciﬁc document representation

Original: [Equation]
Revised: [Equation]

Original: These are in turn aggregated into an overall document representation with class weights
Revised: These are in turn aggregated into an overall document representation with class weights

Original: η c e i =
Revised: η c e i =

Original: C (cid:88)
Revised: C (cid:88)

Original: [Equation]
Revised: [Equation]

Original: This representation will be used by the rule attention submodule to estimate conditional LF reliability.
Revised: This representation will be used by the rule attention submodule to estimate conditional LF reliability.

Original: The class-speciﬁc embeddings
Revised: The class-speciﬁc embeddings

Original: ˜ e
Revised: ˜ e

Original: ( c ) i are also used to compute the document’s class probabilities:
Revised: ( c ) i are also used to compute the document’s class probabilities:

Original: [Equation]
Revised: [Equation]

Original: where p ( c ) i =
Revised: where p ( c ) i =

Original: w ( c ) p T ˜
Revised: w ( c ) p T ˜

Original: e ( c ) i and w ( c ) p is a weight vector corresponding to each class.
Revised: e ( c ) i and w ( c ) p is a weight vector corresponding to each class.

Original: In addition to serving as this submodule’s prediction of the document’s label, these probabilities can additionally serve as measures of the document’s suitability to generate keywords to be used in future LFs.
Revised: In addition to serving as this submodule’s prediction of the document’s label, these probabilities can additionally serve as measures of the document’s suitability to generate keywords to be used in future LFs.

Original: 4.2.2 Rule attention
Revised: 4.2.2 Rule attention

Original: The RuleAttention submodule learns measures of labeling function validity conditioned on the encoded document representation e i .
Revised: The RuleAttention submodule learns measures of labeling function validity conditioned on the encoded document representation e i .

Original: It outputs document-speciﬁc LF reliability scores as well as soft labels to be used in co-training our neural classiﬁer.
Revised: It outputs document-speciﬁc LF reliability scores as well as soft labels to be used in co-training our neural classiﬁer.

Original: We ﬁrst learn conditional LF reliability scores
Revised: We ﬁrst learn conditional LF reliability scores

Original: z i for document d i as
Revised: z i for document d i as

Original: [Equation]
Revised: [Equation]

Original: We weight our masked LFs by these reliability scores using a masked elementwise product s i =
Revised: We weight our masked LFs by these reliability scores using a masked elementwise product s i =

Original: z i (cid:12) (cid:96) i .
Revised: z i (cid:12) (cid:96) i .

Original: This produces probabilistic class labels ˆ
Revised: This produces probabilistic class labels ˆ

Original: y i for each labeled document.
Revised: y i for each labeled document.

Original: We use these soft labels and the class probabilities p
Revised: We use these soft labels and the class probabilities p

Original: i TokenAttention using probabilistic cross entropy loss:
Revised: i TokenAttention using probabilistic cross entropy loss:

Original: [Equation]
Revised: [Equation]

Original: 4.3 Rule Proposal Network
Revised: 4.3 Rule Proposal Network

Original: ReGAL’s rule proposal network (RPN) enables ReGAL to generate new rules given a set of seed rules.
Revised: ReGAL’s rule proposal network (RPN) enables ReGAL to generate new rules given a set of seed rules.

Original: The RPN takes as inputs both the class-conditioned word level attention a ci,t and document-level class probabilities p i and outputs a score
Revised: The RPN takes as inputs both the class-conditioned word level attention a ci,t and document-level class probabilities p i and outputs a score

Original: τ ( c ) j for each v j ∈ V corresponding to how strongly v j represents class c .
Revised: τ ( c ) j for each v j ∈ V corresponding to how strongly v j represents class c .

Original: These scores are calculated as:
Revised: These scores are calculated as:

Original: [Equation]
Revised: [Equation]

Original: Here, γ ∈ [0 , 1] is a parameter that controls how much our ReGAL’s RPN balances between the coverage of a token (i.e. how often it occurs) and its instance level importance.
Revised: Here, γ ∈ [0 , 1] is a parameter that controls how much our ReGAL’s RPN balances between the coverage of a token (i.e. how often it occurs) and its instance level importance.

Original: Low values of γ favor tokens with high coverage while high values of γ favor rules with high impact tokens without regard for coverage.
Revised: Low values of γ favor tokens with high coverage while high values of γ favor rules with high impact tokens without regard for coverage.

Original: Since the types of rules needed may differ as training progresses, we allow users to choose γ for each round of proposed rules.
Revised: Since the types of rules needed may differ as training progresses, we allow users to choose γ for each round of proposed rules.

Original: In practice, we ﬁnd that γ ∈ [0 . 5 , 0 . 9] tend to produce good rules.
Revised: In practice, we ﬁnd that γ ∈ [0 . 5 , 0 . 9] tend to produce good rules.

Original: After calculating token scores τ ( c ) j , the RPN proposes up to k new LFs for each class by choosing the top k scoring tokens { v ( c ) 1 , . .
Revised: After calculating token scores τ ( c ) j , the RPN proposes up to k new LFs for each class by choosing the top k scoring tokens { v ( c ) 1 , . .

Original: . , v ( c ) k } that are not highly scoring for any other class c (cid:48) .
Revised: . , v ( c ) k } that are not highly scoring for any other class c (cid:48) .

Original: These tokens each induce a labeling function of the form HAS(x, v ( c ) i )
Revised: These tokens each induce a labeling function of the form HAS(x, v ( c ) i )

Original: → c , where we assign the class label c to a text x if it contains the token v ( c ) i .
Revised: → c , where we assign the class label c to a text x if it contains the token v ( c ) i .

Original: These proposed LFs are then presented to a human annotator, who can choose to accept or reject them.
Revised: These proposed LFs are then presented to a human annotator, who can choose to accept or reject them.

Original: 4.4 Model optimization
Revised: 4.4 Model optimization

Original: We optimize our entire model by minimizing the unweighted sum of the loss functions of its components:
Revised: We optimize our entire model by minimizing the unweighted sum of the loss functions of its components:

Original: L
Revised: L

Original: [Equation]
Revised: [Equation]

Original:  Experiments and Discussion
Revised:  Experiments and Discussion

Original: We test the ability of ReGAL to generate meaningful, diverse rules to assist humans in efﬁcient data annotation.
Revised: We test the ability of ReGAL to generate meaningful, diverse rules to assist humans in efﬁcient data annotation.

Original: In doing so, we follow many aspects of the experimental setup of [16] and speciﬁcally demonstrate how our model can be used to add to their carefully curated rule-based labeling LFs.
Revised: In doing so, we follow many aspects of the experimental setup of [16] and speciﬁcally demonstrate how our model can be used to add to their carefully curated rule-based labeling LFs.

Original: We note that this a work is in progress and thus models have not undergone hyperparameter optimization.
Revised: We note that this a work is in progress and thus models have not undergone hyperparameter optimization.

Original: However, even without this optimization we demonstrate that ReGAL is both able to generate coherent rules and adeptly denoise rule-induced labels, making it a powerful tool to assist annotators in generating LFs and disambiguate conﬂicted labels.
Revised: However, even without this optimization we demonstrate that ReGAL is both able to generate coherent rules and adeptly denoise rule-induced labels, making it a powerful tool to assist annotators in generating LFs and disambiguate conﬂicted labels.

Original: 5.1 Datasets and tasks
Revised: 5.1 Datasets and tasks

Original: We evaluate our model’s performance on two well-known sentiment classiﬁcation datasets.
Revised: We evaluate our model’s performance on two well-known sentiment classiﬁcation datasets and a topic classiﬁcation dataset.

Original: • Yelp is a collection of Yelp restaurant reviews classiﬁed according to their sentiment [24] • IMDB is a set of movie reviews classiﬁed according to sentiment [25] 5.2 Baselines
Revised: • Yelp is a collection of Yelp restaurant reviews classiﬁed according to their sentiment [24] • IMDB is a set of movie reviews classiﬁed according to sentiment [25] • AGnews is a news corpus for topic classiﬁcation with four classes: sports, technology, politics, and business [26]. Basic summary statistics on our data are found in Table 1. [Table] [Table]

Revised: 5.2 Baselines

Original: We compare the predictive performance of our model to three strong baselines for text classiﬁcation with weak supervision:
Revised: We compare the predictive performance of our model to three strong baselines for text classiﬁcation with weak supervision:

Original: [Table]

Original: • Majority voting of the labeling functions, where ties are broken by choosing the most common class as speciﬁed by the labeling functions.
Revised: • Majority voting of the labeling functions, where ties are broken by choosing the most common class as speciﬁed by the labeling functions.

Original: • BERT [23] is a transformer-based language model that has shown exceptional performance on a wide-range of NLP tasks.
Revised: • BERT [23] is a transformer-based language model that has shown exceptional performance on a wide-range of NLP tasks.

Original: We train BERT with the majority vote labels from our LFs.
Revised: We train BERT with the majority vote labels from our LFs.

Revised: • WeSTClass

Revised: [15] is a weakly supervised text classiﬁcation that trains a classiﬁer using documents generated using user-provided labeling rules.

Original: • Epoxy [26] is a recent weak supervision paradigm that uses combined pretrained embeddings with anchored weakly-labeled examples to enable interactive model training.
Revised: • Epoxy [27] is a recent weak supervision paradigm that uses combined pretrained embeddings with anchored weakly-labeled examples to enable interactive model training.

Original: • MSWS [16] is a denoising method for multi-source weak supervision that co-trains a rule denoiser with a neural classiﬁer to learn optimal weightings for rules and label unmatched samples.
Revised: • MSWS [16] is a denoising method for multi-source weak supervision that co-trains a rule denoiser with a neural classiﬁer to learn optimal weightings for rules and label unmatched samples.

Original: • Fully Supervised  We train a BERT model with a fully connected layer to provide baseline performance for a model trained with ground-truth labels.
Revised: • Fully Supervised BERT We train a BERT model with a fully connected layer to provide baseline performance for a model trained with ground-truth labels.

Original: 5.3 Rule Denoising
Revised: 5.3 Rule Denoising

Original: Parsing signal from noise is critical to learning from weak and rule-based supervision.
Revised: Parsing signal from noise is critical to learning from weak and rule-based supervision.

Original: Accordingly, we compare ReGAL’s ability to that of our baselines in accurately classifying instances based on a set of seed rules, which are shown in Table 1.
Revised: Accordingly, we compare ReGAL’s ability to that of our baselines in accurately classifying instances based on a set of seed rules, which are shown in Table 2.

Original: We provided exactly each classwith exactly one labeling function consisting of  six  keywords or phrases adapted from [16].
Revised: We provided a single seed rule for each class of Yelp and IMDB and six single-word LFs for each class of AG-News, where LFs contain keywords or phrases adapted from [16].

Original: If any of these keywords is found in document d
Revised: If any of these keywords is found in document d

Original: i , the LF assigns its label; otherwise, it abstains from labeling d i .
Revised: i , the LF assigns its label; otherwise, it abstains from labeling d i .

Original: After providing seed rules [Table] we trained ReGAL with learning rate 0 .
Revised: After providing seed rules, we trained ReGAL with learning rate 0 .

Original: 001 and batch size 64, generating new rules every 100 batches.
Revised: 001 and batch size 64, generating new rules every 100 batches for Yelp and IMDB and every 250 batches for AGNews.

Original: We continued  iterating until we saw multiple consecutive rounds with for which at least one class proposed no high-quality rules.
Revised: We continued [Table] [Table] iterating until we saw multiple consecutive rounds with for which at least one class proposed no high-quality rules.

Revised: We calculate rule proposal weights by aggregating word attention over the entire dataset for Yelp and IMDB and over a randomly sampled 1/6 of AGNews.

Revised: A summary of important hyperparameter choices is given in Table 1.

Original: In each case, proposed rules were adjudicated by university students to assess their utility and relevance.
Revised: Proposed rules were adjudicated by university students to assess their utility and relevance.

Original: One factor that majorly differentiates ReGAL from our baselines is its ability to interactively generate labeling functions to expand label coverage.
Revised: One factor that majorly differentiates ReGAL from our baselines is its ability to interactively generate labeling functions to expand label coverage.

Original: This potentially gives ReGAL an advantage over our baselines which cannot generate new labeling rules.
Revised: This potentially gives ReGAL an advantage over our baselines which cannot generate new labeling rules.

Original: Accordingly, we show the performance of each of these models using (1) only seed LFs and (2) seed + generated LFs from ReGAL.
Revised: Accordingly, we show the performance of each of these models using (1) only seed LFs and (2) seed + generated LFs from ReGAL.

Original: The results are shown in Table 2.
Revised: The results are shown in Table 3.

Revised: When rule generation persists for many rounds (as is the case with

Revised: AGNews), ReGAL’s underlying model sometimes overﬁts to previously generated rules which results in performance degradation.

Revised: We accordingly retrain the AGNews rule denoising module with probabilistic labels after solidifying labeling rules, which converges in just 250 batches.

Original: Interestingly, although ReGAL generates coherent and intuitive rules (see section 5.4), these rules actually degrade ReGAL’s performance.
Revised: Interestingly, although ReGAL generates coherent and intuitive rules (see section 5.4), these rules actually degrade ReGAL’s performance.

Original: This is particularly surprising considering that the added rules substantially boost the performance of each of the baselines.
Revised: This is particularly surprising considering that the added rules substantially boost the performance of each of the baselines.

Original: We suspect that this may be due to the masking of rule keywords in the encoder, which could inhibit the model from learning their signal.
Revised: We suspect that this may be due to the masking of rule keywords in the encoder, which could inhibit the model from learning their signal.

Original: We will investigate this anomaly further as we continue to develop the model.
Revised: We will investigate this anomaly further as we continue to develop the model.

Original: 5.4 Evaluating Generated Rules
Revised: 5.4 Evaluating Generated Rules

Original: We evaluate the ability of our model to identify promising rules by providing it a small number of seed rules and qualitatively evaluating its ability to discover additional high-quality rules.
Revised: We evaluate the ability of our model to identify promising rules by providing it a small number of seed rules and qualitatively evaluating its ability to discover additional high-quality rules.

Original: In tables 3 and 4, we display the top 5 generated rules from the ﬁrst two iterations of ReGAL for Yelp and IMDB respectively.
Revised: In Tablewe display the top 3 generated rules from the ﬁrst two iterations of ReGAL for Yelp and in Tablewe show rules from iterations 1,2, and 5 on AG News.

Original: From these results, one can see that on the ﬁrst iteration, ReGAL generates a very high proportion of highly informative rules.
Revised: From these results, one can see that on the ﬁrst iteration, ReGAL generates a very high proportion of highly informative rules.

Original: For example, ReGAL’s ﬁrst set of rules touches on multiple diverse aspects of a restaurant’s quality, including food quality, speed, and friendliness of staff.
Revised: For example, ReGAL’s ﬁrst set of rules touches on multiple diverse aspects of a restaurant’s quality, including food quality, speed, and friendliness of staff.

Original: IMDB’s ﬁrst-iteration

Original: [Table] [Table] rules capture similarly strong signal, picking up that negative reviews call movies “boring” and “dull” while positive ones cite them as “touching” or “marvelous. [Table]
Revised: AGNews shows similar patterns, picking up on elections, stock trading, soccer leagues, and tech companies in the ﬁrst iteration of rules.

Original: Unfortunately, rule quality appears to degrade sharply after the ﬁrst iteration.
Revised: Unfortunately, rule quality appears to degrade sharply after the ﬁrst iteration.

Original: After a 90% acceptance rate in the ﬁrst round of generation on Yelp, ReGAL failed to generate a single accepted rule in round2.
Revised: After a 90% acceptance rate in the ﬁrst round of generation on Yelp, ReGAL failed to generate a single accepted rule in round

Original: While the results on IMDB were somewhat more promising, ReGAL only generated meaningful rules  for a single class in the second round, which could potentially lead to label function imbalance and thus improper ﬁtting.
Revised: Rules generated on AGNews were slightly more promising, with coherent rules persisting into later rounds, though there is clear quality degradation for speciﬁc classes. One clear instance of this [Table]

Original: Because ReGAL is an active learning framework
Revised: is with the persistent proposal of rules starting with ‘war’ (one of the seed rules)

Original: [Table]
Revised: [Table]

Original: users are able toevaluate proposed rules and control how the model expands its labeling space. This makes ReGAL resiliant to spurious generated rules and allows users to use it to quickly and meaningfully expand their rule space
Revised: which demonstrates overﬁtting to the keyword-based labeling functions. A similar pattern can be seen with nuclear arms development keywords (e

Original: [Table]
Revised: [Table]

Revised: g. ‘enrichment’ in iteration 5) caused by the acceptance of ‘nuclear’ in the tech category at iteration 3.

Revised: Because ReGAL is an active learning framework

Original: [Table]
Revised: [Table]

Revised: users are able to evaluate proposed rules and control how the model expands its labeling space.

Revised: This makes ReGAL resiliant to spurious generated rules and allows users to use it to quickly and meaningfully expand their rule space.

Original:  Future Work
Revised:  Future Work

Original: As we continue to develop ReGAL, we intend to make improvements to both our model and evaluation.
Revised: As we continue to develop ReGAL, we intend to make improvements to both our model and evaluation.

Original: First, we will add a more diverse set of baselines and evaluation datasets that allow us to compare
Revised: First, we will add a more diverse set of baselines and evaluation datasets that allow us to compare

Original: ReGAL to both traditional active learning frameworks as well as interactive data programming paradigms [12].
Revised: ReGAL to both traditional active learning frameworks as well as interactive data programming paradigms [12].

Original: These, in turn, will be tested on a more diverse array of text classiﬁcation tasks, especially those requiring domain expertise (e.g. biomedical).
Revised: These, in turn, will be tested on a more diverse array of text classiﬁcation tasks, especially those requiring domain expertise (e.g. biomedical).

Original: Second, we will expand ReGAL’s rule generating capacity to encompass multi-token sequences (e.g. for keywords that are split into multiple tokens) or non-contiguous textual patterns.
Revised: Second, we will expand ReGAL’s rule generating capacity to encompass multi-token sequences (e.g. for keywords that are split into multiple tokens) or non-contiguous textual patterns.

Original: Finally, we will conduct user tests to determine and optimize the quality of human interactions to make ReGAL more user-friendly and accurate.
Revised: Finally, we will conduct user tests to determine and optimize the quality of human interactions to make ReGAL more user-friendly and accurate.

Original: One promising area for future work is adapting ReGAL to work with crowdsourced labeling.
Revised: One promising area for future work is adapting ReGAL to work with crowdsourced labeling.

Original: In such a framework, LFs could be adopted only after agreement from multiple annotators, which would lead to higher overall rule quality.
Revised: In such a framework, LFs could be adopted only after agreement from multiple annotators, which would lead to higher overall rule quality.

Original: We forsee that this could natturally lead to a two-stage crowdsourcing framework in which earlier annotators would focus on broadening coverage with new LFs while subsequent annotators could with higher LF uncertainty (e.g. due to conﬂicting LF matches).
Revised: We forsee that this could natturally lead to a two-stage crowdsourcing framework in which earlier annotators would focus on broadening coverage with new LFs while subsequent annotators could with higher LF uncertainty (e.g. due to conﬂicting LF matches).

Original: This would increase crowdsourcing efﬁciency by focusing annotators on providing instancelevel supervision on difﬁcult samples while still producing rule-induced labels on the remainder of the data.
Revised: This would increase crowdsourcing efﬁciency by focusing annotators on providing instancelevel supervision on difﬁcult samples while still producing rule-induced labels on the remainder of the data.

Original:  Conclusion
Revised:  Conclusion

Original: In this work, we presented our work-in-progress ReGAL, a text classiﬁcation model capable of both denoising multiple sources of noisy supervision and generating high-quality labeling functions capable of inductively labeling uncovered portions of dataset.
Revised: In this work, we presented our work-in-progress ReGAL, a text classiﬁcation model capable of both denoising multiple sources of noisy supervision and generating high-quality labeling functions capable of inductively labeling uncovered portions of dataset.

Original: In doing so, ReGAL offers the potential to substantially reduce human labeling efforts by helping them identify quality labeling rules they would otherwise miss.
Revised: In doing so, ReGAL offers the potential to substantially reduce human labeling efforts by helping them identify quality labeling rules they would otherwise miss.

Original: We demonstrate ReGAL’s utility by showing that it both generates coherent, reasonable rules and also that using the selected labeling rules improves the performance of each of our baseline models.
Revised: We demonstrate ReGAL’s utility by showing that it both generates coherent, reasonable rules and also that using the selected labeling rules improves the performance of each of our baseline models.

Original: Broader Impact
Revised: Broader Impact

Original: This work has major potential to accelerate advancements in low-resource NLP domains by assisting subject matter experts in discovering patterns capable of quickly labeling their data.
Revised: This work has major potential to accelerate advancements in low-resource NLP domains by assisting subject matter experts in discovering patterns capable of quickly labeling their data.

Original: This would allow them to tackle more problems for which they previously lacked data while simultaneously reducing the cost of doing so.
Revised: This would allow them to tackle more problems for which they previously lacked data while simultaneously reducing the cost of doing so.

Original: One could observe similar beneﬁts in NLP for scientiﬁc and biomedical texts.
Revised: One could observe similar beneﬁts in NLP for scientiﬁc and biomedical texts.

Original: The breadth of even a single scientiﬁc discipline is so broad that a single SME may miss key semantic indicators that are not associated with his/her subﬁeld.
Revised: The breadth of even a single scientiﬁc discipline is so broad that a single SME may miss key semantic indicators that are not associated with his/her subﬁeld.

Original: Thus, automatic generation of labeling rules can potentially increase the diversity of scientiﬁc and biomedical NLP tasks and thus accelerate the discovery of new insights derived from text mining.
Revised: Thus, automatic generation of labeling rules can potentially increase the diversity of scientiﬁc and biomedical NLP tasks and thus accelerate the discovery of new insights derived from text mining.

Original: There are two potential negative impacts of this work, both of which hinge on the cognitive biases of humans involved in data annotation.
Revised: There are two potential negative impacts of this work, both of which hinge on the cognitive biases of humans involved in data annotation.

Original: First, our model exploits patterns in seed rules for generating new labeling functions, which may lead it to miss important labeling signals that are independent from the given seed LFs.
Revised: First, our model exploits patterns in seed rules for generating new labeling functions, which may lead it to miss important labeling signals that are independent from the given seed LFs.

Original: Second, the reduction of human effort in creating LFs may lead to annotators accepting proposed rule without sufﬁciently rigorous evaluation, thus degrading LF quality.
Revised: Second, the reduction of human effort in creating LFs may lead to annotators accepting proposed rule without sufﬁciently rigorous evaluation, thus degrading LF quality.

Original: Acknowledgments and Disclosure of Funding
Revised: Acknowledgments and Disclosure of Funding

Original: This work was supported by National Science Foundation CAREER grant 1944247, National Institute of Health grant R21-CA232249, Whitaker Biomedical Engineering Seed Grant for COVID-19,
Revised: This work was supported by National Science Foundation CAREER grant 1944247, National Institute of Health grant R21-CA232249, Whitaker Biomedical Engineering Seed Grant for COVID-19,

Original: Alzheimer’s Association Research Grant Award, Childrens Hospital of Atlanta Aﬂac Pilot Grant
Revised: Alzheimer’s Association Research Grant Award, Childrens Hospital of Atlanta Aﬂac Pilot Grant

Original: Award, National Science Foundation award III-2008334, an Amazon Faculty Award, and a Google
Revised: Award, National Science Foundation award III-2008334, an Amazon Faculty Award and a Google

Original: Faculty Award.
Revised: Faculty Award.