Original: L EARNING FROM E XPLANATIONS WITH N EURALM ODULE E XECUTION T REE
Revised: L EARNING FROM E XPLANATIONS WITH N EURAL E XECUTION T REE

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: While deep neural networks have achieved impressive performance on a range of NLP tasks, these data-hungry models heavily rely on labeled data.
Revised: While deep neural networks have achieved impressive performance on a range of NLP tasks, these data-hungry models heavily rely on labeled data.

Original: To make the most of each example, previous work has introduced natural language (NL) explanations to serve as supplements to mere labels.
Revised: To make the most of each example, previous work has introduced natural language (NL) explanations to serve as supplements to mere labels.

Original: Such NL explanations can provide sufﬁcient domain knowledge for generating more labeled data over new instances, while the annotation time only doubles.
Revised: Such NL explanations can provide sufﬁcient domain knowledge for generating more labeled data over new instances, while the annotation time only doubles.

Original: However, directly applying the NL explanations for augmenting model learning encounters two challenges.
Revised: However, directly applying the NL explanations for augmenting model learning encounters two challenges.

Original: First, NL explanations are unstructured and inherently compositional, which asks for a modularized model to represent their semantics.
Revised: First, NL explanations are unstructured and inherently compositional, which asks for a modularized model to represent their semantics.

Original: Second, NL explanations often have large numbers of linguistic variants, resulting in low recall and limited generalization ability when applied to unlabeled data.
Revised: Second, NL explanations often have large numbers of linguistic variants, resulting in low recall and limited generalization ability when applied to unlabeled data.

Original: In this paper, we propose a novel NeuralModular Execution Tree (NMET) framework for augmenting sequence classiﬁcation with NL explanations.
Revised: In this paper, we propose a novel Neural Execution Tree (NExT) framework for augmenting sequence classiﬁcation with NL explanations.

Original: First, we transform NL explanations into executable logical forms using a semantic parser.
Revised: First, we transform NL explanations into executable logical forms using a semantic parser.

Original: NMET then employs a neural module network architecture to generalize the different types of actions (speciﬁed by the logical forms) for labeling data instances and accumulates the results with soft logic, which substantially increases the coverage of each NL explanation.
Revised: NExT then employs a neural module network architecture to generalize the different types of actions (speciﬁed by the logical forms) for labeling data instances and accumulates the results with soft logic, which substantially increases the coverage of each NL explanation.

Original: Experiments on two NLP tasks, relation extraction, and sentiment analysis, demonstrate its superiority over baseline methods by leveraging NL explanation.
Revised: Experiments on two NLP tasks, relation extraction, and sentiment analysis, demonstrate its superiority over baseline methods by leveraging NL explanation.

Original: Its extension to multi-hop question answering achieves performance gain with light annotation effort.
Revised: Its extension to multi-hop question answering achieves performance gain with light annotation effort.

Original: Also, NMET achieves much better performance compared to traditional label-only supervised models in the same annotation time.
Revised: Also, NExT achieves much better performance compared to traditional label-only supervised models in the same annotation time.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Deep neural networks have achieved state-of-the-art performance on a wide range of natural language processing tasks.
Revised: Deep neural networks have achieved state-of-the-art performance on a wide range of natural language processing tasks.

Original: However, they usually require massive labeled data, which restricts their applications in scenarios where data annotation is expensive.
Revised: However, they usually require massive labeled data, which restricts their applications in scenarios where data annotation is expensive.

Original: The traditional way of providing supervision is human-generated labels.
Revised: The traditional way of providing supervision is human-generated labels.

Original: For example, in sentiment analysis, given a sentence “Quality ingredients preparation all around, and a very fair price for NYC” , an annotator should label it as “Positive” .
Revised: For example, in sentiment analysis, given a sentence “Quality ingredients preparation all around, and a very fair price for NYC” , an annotator should label it as “Positive” .

Original: However, the label itself does not provide information about how the decision is made.
Revised: However, the label itself does not provide information about how the decision is made.

Original: A more informative method is to allow annotators to explain their decisions in natural language so that the annotation can generalize to other examples.
Revised: A more informative method is to allow annotators to explain their decisions in natural language so that the annotation can generalize to other examples.

Original: In the above example, an explanation can be “Positive, because the word price is directly preceded by fair” , which can generalize to instances like “Delicious food with a fair price” .
Revised: In the above example, an explanation can be “Positive, because the word price is directly preceded by fair” , which can generalize to instances like “Delicious food with a fair price” .

Original: Natural language (NL) explanations have shown effectiveness in providing additional supervision, especially in low-resource settings (Srivastava et al., 2017;
Revised: Natural language (NL) explanations have shown effectiveness in providing additional supervision, especially in low-resource settings (Srivastava et al., 2017;

Original: Hancock et al., 2018).
Revised: Hancock et al., 2018).

Original: Also, they can be easily collected from human annotators without signiﬁcantly increasing their annotation efforts.
Revised: Also, they can be easily collected from human annotators without signiﬁcantly increasing their annotation efforts.

Original: However, exploiting NL explanations as supervision is challenging due to the complex nature of human languages.
Revised: However, exploiting NL explanations as supervision is challenging due to the complex nature of human languages.

Original: First of all, textual data are not well-structured, and thus we have to parse explanations into logical forms for machine to better utilize them.
Revised: First of all, textual data are not well-structured, and thus we have to parse explanations into logical forms for machine to better utilize them.

Original: Also, linguistic variants are ubiquitous, which makes it difﬁcult to generalize an NL explanation for matching sentences that are semantically equivalent but having different word usage.
Revised: Also, linguistic variants are ubiquitous, which makes it difﬁcult to generalize an NL explanation for matching sentences that are semantically equivalent but having different word usage.

Original: When we perform exact matching with the previous example explanation, it can fail to annotate sentences with “reasonable price” or “good deal”.
Revised: When we perform exact matching with the previous example explanation, it can fail to annotate sentences with “reasonable price” or “good deal”.

Original: [Figure]
Revised: [Figure]

Original: Attempts have been made to train classiﬁers with NL explanations.
Revised: Attempts have been made to train classiﬁers with NL explanations.

Original: Srivastava et al.
Revised: Srivastava et al.

Original: (2017) use NL explanations as additional features of data.
Revised: (2017) use NL explanations as additional features of data.

Original: They map explanations to logical forms with a semantic parser and use them to generate binary features for all instances.
Revised: They map explanations to logical forms with a semantic parser and use them to generate binary features for all instances.

Original: Hancock et al.
Revised: Hancock et al.

Original: (2018) employs a rule-based semantic parser to get logical forms (i.e. “labeling function”) from NL explanations that generate noisy labeled datasets used for training models.
Revised: (2018) employs a rule-based semantic parser to get logical forms (i.e. “labeling function”) from NL explanations that generate noisy labeled datasets used for training models.

Original: While both methods claim huge performance improvements, they neglect the importance of linguistic variants, thus resulting in a very low recall.
Revised: While both methods claim huge performance improvements, they neglect the importance of linguistic variants, thus resulting in a very low recall.

Original: Also, their methods of evaluating explanations on new instances are oversimpliﬁed (e.g. comparison/logic operators), making their methods over-conﬁdent.
Revised: Also, their methods of evaluating explanations on new instances are oversimpliﬁed (e.g. comparison/logic operators), making their methods over-conﬁdent.

Original: In the above example, a sentence like “Decent sushi at a fair enough price” will be rejected because of the “directly preceded” requirement.
Revised: In the above example, a sentence like “Decent sushi at a fair enough price” will be rejected because of the “directly preceded” requirement.

Original: Therefore, we believe that the generalization ability of NL explanations is under-explored.
Revised: Therefore, we believe that the generalization ability of NL explanations is under-explored.

Original: We emphasize that a good data annotation method should 1) be able to generalize annotations to semantically similar instances (beyond stemming, part-of-speech, etc.) and 2) model the uncertainty in annotations.
Revised: We emphasize that a good data annotation method should 1) be able to generalize annotations to semantically similar instances (beyond stemming, part-of-speech, etc.) and 2) model the uncertainty in annotations.

Original: Towards these goals, we propose NeuralModular Execution Tree (NMET) framework for learning neural models with explanations, as illustrated in Figure 1.
Revised: Towards these goals, we propose Neural Execution Tree (NExT) framework for learning neural models with explanations, as illustrated in Figure 1.

Original: Given a raw corpus and a set of NL explanations, we ﬁrst parse the NL explanations to machine-actionable logic forms by a combinatory categorial grammar (CCG) based semantic parser.
Revised: Given a raw corpus and a set of NL explanations, we ﬁrst parse the NL explanations to machine-actionable logic forms by a combinatory categorial grammar (CCG) based semantic parser.

Original: Different from previous work, we “soften” the annotation process by generalizing the predicates using neural module network and changing the labeling process from exact matching to fuzzy matching.
Revised: Different from previous work, we “soften” the annotation process by generalizing the predicates using neural module network and changing the labeling process from exact matching to fuzzy matching.

Original: We introduce four types of matching modules, namely String Matching Module , Soft Counting Module , Logical Calculation Module , and Deterministic Function Module , which serve as matching semantically similar words, scoring different positions, accumulating matching scores from other modules, and dealing with deterministic predicates, respectively.
Revised: We introduce four types of matching modules, namely String Matching Module , Soft Counting Module , Logical Calculation Module , and Deterministic Function Module , which serve as matching semantically similar words, scoring different positions, accumulating matching scores from other modules, and dealing with deterministic predicates, respectively.

Original: We calculate the matching scores and ﬁnd for each instance the most similar logical form.
Revised: We calculate the matching scores and ﬁnd for each instance the most similar logical form.

Original: Thus, all instances in the raw corpus can be assigned a label and used to train the neural model.
Revised: Thus, all instances in the raw corpus can be assigned a label and used to train the neural model.

Original: We conduct extensive experiments on two representative tasks, relation extraction, and sentiment analysis.
Revised: We conduct extensive experiments on two representative tasks, relation extraction, and sentiment analysis.

Original: Experimental results demonstrate the superiority of NMET over various baseline methods.
Revised: Experimental results demonstrate the superiority of NExT over various baseline methods.

Original: Also, we adapted NMET for multi-hop question answering task, in which it achieves performance improvement with only 21 explanations and 5 rules.
Revised: Also, we adapted NExT for multi-hop question answering task, in which it achieves performance improvement with only 21 explanations and 5 rules.

Original:  L EARNING TO A UGMENT S EQUENCE M ODELS WITH NL E XPLANATIONS
Revised:  L EARNING TO A UGMENT S EQUENCE M ODELS WITH NL E XPLANATIONS

Original: We consider the task of training classiﬁers with natural language explanations for text classiﬁcation (e.g., relation extraction, sentiment analysis) in a low-resource setting.
Revised: We consider the task of training classiﬁers with natural language explanations for text classiﬁcation (e.g., relation extraction, sentiment analysis) in a low-resource setting.

Original: Speciﬁcally, given a raw corpus S = { x i } Ni =1 ⊆ X and a predeﬁned label set Y , our goal is to learn a classiﬁer f c : X → Y .
Revised: Speciﬁcally, given a raw corpus S = { x i } Ni =1 ⊆ X and a predeﬁned label set Y , our goal is to learn a classiﬁer f c : X → Y .

Original: We ask human annotators to view a subset S (cid:48) of the corpus S and provide for each instance x ∈ S (cid:48) an explanation e and a label y , which explains why the instance should receive that label.
Revised: We ask human annotators to view a subset S (cid:48) of the corpus S and provide for each instance x ∈ S (cid:48) an explanation e and a label y , which explains why the instance should receive that label.

Original: Note that |S (cid:48) | (cid:28) |S| , which requires our framework to learn with very limited human supervision.
Revised: Note that |S (cid:48) | (cid:28) |S| , which requires our framework to learn with very limited human supervision.

Original: Approach Overview.
Revised: Approach Overview.

Original: We develop a multi-stage learning framework to leverage NL explanations in a weakly-supervised setting.
Revised: We develop a multi-stage learning framework to leverage NL explanations in a weakly-supervised setting.

Original: An overview of our framework is depicted in Fig.
Revised: An overview of our framework is depicted in Fig.

Original: Our NMET framework consists of three stages, namely explanation parsing stage, dataset partition stage, and model learning stage.
Revised: Our NExT framework consists of three stages, namely explanation parsing stage, dataset partition stage, and model learning stage.

Original: Explanation Parsing.
Revised: Explanation Parsing.

Original: To leverage the unstructured human explanations E = { e j } |S (cid:48)
Revised: To leverage the unstructured human explanations E = { e j } |S (cid:48)

Original: | j =1 , we turn them into machine-actionable logical forms (i.e., labeling functions)
Revised: | j =1 , we turn them into machine-actionable logical forms (i.e., labeling functions)

Original: (Ratner et al., 2016), which can be denoted as F = { f j : X → { 0 , 1 }} |S (cid:48)
Revised: (Ratner et al., 2016), which can be denoted as F = { f j : X → { 0 , 1 }} |S (cid:48)

Original: | j =1 , where 1 indicates the logical form matches the input sequence and 0 otherwise.
Revised: | j =1 , where 1 indicates the logical form matches the input sequence and 0 otherwise.

Original: To access the labels, we introduce a function h : F → Y that maps each logical form f j to the label y j of its explanation e j .
Revised: To access the labels, we introduce a function h : F → Y that maps each logical form f j to the label y j of its explanation e j .

Original: Examples are given in Fig.
Revised: Examples are given in Fig.

Original: We use
Revised: We use

Original: Combinatory Categorial Grammar (CCG) based semantic parsing (Zettlemoyer & Collins, 2012;
Revised: Combinatory Categorial Grammar (CCG) based semantic parsing (Zettlemoyer & Collins, 2012;

Original: Artzi et al., 2015), an approach that couples syntax with semantics, to convert each NL explanation e j to a logical form f j .
Revised: Artzi et al., 2015), an approach that couples syntax with semantics, to convert each NL explanation e j to a logical form f j .

Original: Following Srivastava et al.
Revised: Following Srivastava et al.

Original: (2017), we ﬁrst compile a domain lexicon that maps each word to its syntax and logical predicate.
Revised: (2017), we ﬁrst compile a domain lexicon that maps each word to its syntax and logical predicate.

Original: Frequently-used predicates are listed in the Appendix.
Revised: Frequently-used predicates are listed in the Appendix.

Original: For each explanation, the parser can generate many possible logical forms based on CCG grammar.
Revised: For each explanation, the parser can generate many possible logical forms based on CCG grammar.

Original: To identify the correct one from these logical forms, we use a feature vector φ ( f ) ∈ R d with each element counting the number of applications of a particular CCG combinator (similar to Zettlemoyer & Collins (2007)).
Revised: To identify the correct one from these logical forms, we use a feature vector φ ( f ) ∈ R d with each element counting the number of applications of a particular CCG combinator (similar to Zettlemoyer & Collins (2007)).

Original: Speciﬁcally, given an explanation e i , the semantic parser parameterized by θ ∈ R d outputs a probability distribution over all possible logical forms Z e i .
Revised: Speciﬁcally, given an explanation e i , the semantic parser parameterized by θ ∈ R d outputs a probability distribution over all possible logical forms Z e i .

Original: The probability of a feasible logical form can be calculated as:
Revised: The probability of a feasible logical form can be calculated as:

Original: [Equation]
Revised: [Equation]

Original: To learn θ , we maximize the probability of y i given e i calculated by marginalizing over all logical forms that match x
Revised: To learn θ , we maximize the probability of y i given e i calculated by marginalizing over all logical forms that match x

Original: i (similar to Liang et al.
Revised: i (similar to Liang et al.

Original: Formally, the objective function is deﬁned as:
Revised: Formally, the objective function is deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: S (cid:48) | (cid:88)
Revised: S (cid:48) | (cid:88)

Original: i = log (cid:0) (cid:88) f : f ( x i )
Revised: i = log (cid:0) (cid:88) f : f ( x i )

Original: =
Revised: =

Original: 1 ∧ h ( f )=
Revised: 1 ∧ h ( f )=

Original: y
Revised: y

Original: i
Revised: i

Original: P θ ( f | e i ) (cid:1) .
Revised: P θ ( f | e i ) (cid:1) .

Original: When the optimal θ ∗ is derived using gradient-based method, the parsing result for e i is deﬁned as f i = arg max f P θ ∗ ( f | e i ) .
Revised: When the optimal θ ∗ is derived using gradient-based method, the parsing result for e i is deﬁned as f i = arg max f P θ ∗ ( f | e i ) .

Original: Dataset Partition.
Revised: Dataset Partition.

Original: After we parse explanations { e i } |S (cid:48) | i =1 into F = { f i } |S (cid:48) | i =1 where each f i corresponds to e i , we use F to ﬁnd exact matches in S and pair them with the corresponding labels.
Revised: After we parse explanations { e i } |S (cid:48) | i =1 into F = { f i } |S (cid:48) | i =1 where each f i corresponds to e i , we use F to ﬁnd exact matches in S and pair them with the corresponding labels.

Original: We denote the number of instances getting labeled by exact matching as N a .
Revised: We denote the number of instances getting labeled by exact matching as N a .

Original: As a result, S is partitioned into a labeled dataset S a = { ( x ai , y ai ) } N a i =1 , and an unlabeled dataset S u = S − { x ai } N a
Revised: As a result, S is partitioned into a labeled dataset S a = { ( x ai , y ai ) } N a i =1 , and an unlabeled dataset S u = S − { x ai } N a

Original: i
Revised: i

Original: = =
Revised: = =

Original: { x uj } N u j = where N u = |S| − N a .
Revised: { x uj } N u j = where N u = |S| − N a .

Original: Model Learning with NeuralModule Execution Tree.
Revised: Model Learning with Neural Execution Tree.

Original: So far we can already make use of exactmatched S a to train a classiﬁer; however, informative instances in S u are left untouched.
Revised: So far we can already make use of exact-matched S a to train a classiﬁer; however, informative instances in S u are left untouched.

Original: We propose 
Revised: We propose Neural

Original: Neural Module Execution Tree (NMET), which relaxes constraints in each logic form f j and substantially improve the rule coverage in S u .
Revised: Execution Tree (NExT), which relaxes constraints in each logic form f j and substantially improve the rule coverage in S u .

Original: Unlabeled instances are then assigned with pseudo labels accompanied by conﬁdence scores.
Revised: Unlabeled instances are then assigned with pseudo labels accompanied by conﬁdence scores.

Original: Classiﬁers will beneﬁt from these soft-matched and pseudolabeled instances.
Revised: Classiﬁers will beneﬁt from these soft-matched and pseudo-labeled instances.

Original: Trainable parameters in NMET are also jointly optimized with the classiﬁer.
Revised: Trainable parameters in NExT are also jointly optimized with the classiﬁer.

Original: Details of NMET and joint training will be introduced in the next section.
Revised: Details of NExT and joint training will be introduced in the next section.

Original:  N EURALM ODULE E XECUTION T REE
Revised:  N EURAL E XECUTION T REE

Original: Given a logical form f and a sentence x , NeuralModule Execution Tree (NMET) will output a matching score u s ∈ [0 , 1] , which indicates how likely the sentence x satisﬁes the logical form f and thus should be given the corresponding label h ( f ) .
Revised: Given a logical form f and a sentence x , Neural Execution Tree (NExT) will output a matching score u s ∈ [0 , 1] , which indicates how likely the sentence x satisﬁes the logical form f and thus should be given the corresponding label h ( f ) .

Original: Speciﬁcally, NMET comprises of four modules to deal with four categories of predicates, namely String Matching Module , Soft Counting 
Revised: Speciﬁcally, NExT comprises of four modules to deal with four categories of predicates, namely String Matching Module , Soft Counting Module ,

Original: Module , Deterministic Function Module , and Logical Calculation Module .
Revised: Deterministic Function Module , and Logical Calculation Module .

Original: Any complex logical form can be disassembled into clauses containing these four categories of predicates.
Revised: Any complex logical form can be disassembled into clauses containing these four categories of predicates.

Original: The four modules are then used to evaluate each clause and then the whole logical form in a softened way.
Revised: The four modules are then used to evaluate each clause and then the whole logical form in a softened way.

Original: Fig.
Revised: Fig.

Original: 2 shows how NMET builds the execution tree from an NL explanation and how it evaluates an unlabeled sentence.
Revised: 2 shows how NExT builds the execution tree from an NL explanation and how it evaluates an unlabeled sentence.

Original: We show in the ﬁgure the corresponding module for each predicate in the logical form.
Revised: We show in the ﬁgure the corresponding module for each predicate in the logical form.

Original: We introduce the four modules respectively as follows.
Revised: We introduce the four modules respectively as follows.

Original: SUBJECTwas murdered on OBJECT 0.3, 0.2, 0.9, 0.2, 0.
Revised: SUBJECTwas murdered on OBJECT 0.3, 0.2, 0.9, 0.2, 0.

Original: The words “who died” precede OBJECT by no more than three words and occur between SUBJECT and OBJECT 0, 1, 1, 1, 0 0.3, 0.2, 0.9, 0.2, 0.
Revised: The words “who died” precede OBJECT by no more than three words and occur between SUBJECT and OBJECT 0, 1, 1, 1, 0 0.3, 0.2, 0.9, 0.2, 0.

Original: (Word(who died) Is Between(SUBJECT, OBJECT))
Revised: (Word(who died) Is Between(SUBJECT, OBJECT))

Original: And (Word(who died)
Revised: And (Word(who died)

Original: Is AtMost(Left(OBJECT) , Num(3 Tokens)))
Revised: Is AtMost(Left(OBJECT) , Num(3 Tokens)))

Original: [Figure]
Revised: [Figure]

Original: Logical Calculation

Original: Soft Counting
Revised: Soft [Figure]

Original: Logical Calculation
Revised: Logical Calculation

Original: String Matching
Revised: String Matching

Original: Logical Calculation DeterministicFunction
Revised: Logical Calculation DeterministicFunction

Original: Date_of_Death
Revised: Date_of_Death

Original: Logical Form:
Revised: Logical [Figure]

Original: Explanation:
Revised: Explanation:

Original: Sentence:
Revised: [Figure]

Original: 3. M ODULES IN NMET
Revised: 3. M ODULES IN NE X T

Original: String Matching Module.
Revised: String Matching Module.

Original: Given a keyword query q derived from an explanation and an input sequence x = [ w 1 , w 2 , ..., w n ] , the string matching module f s ( x , q ) returns a sequence of scores [ s 1 , s 2 , ..., s n ] indicating the similarity between each token w i and the query q .
Revised: Given a keyword query q derived from an explanation and an input sequence x = [ w 1 , w 2 , ..., w n ] , the string matching module f s ( x , q ) returns a sequence of scores [ s 1 , s 2 , ..., s n ] indicating the similarity between each token w i and the query q .

Original: Previous work implements this operation by exact keyword searching, while we augment the module with neural networks to enable capturing semantically similar words.
Revised: Previous work implements this operation by exact keyword searching, while we augment the module with neural networks to enable capturing semantically similar words.

Original: Inspired by Li et al.
Revised: Inspired by Li et al.

Original: (2018), for token w i , we ﬁrst generate N c contexts by sliding windows of various lengths.
Revised: (2018), for token w i , we ﬁrst generate N c contexts by sliding windows of various lengths.

Original: For example, if the maximum window size is 2, the contexts c
Revised: For example, if the maximum window size is 2, the contexts c

Original: i 0 , c i 1 , c i 2 of token w
Revised: i 0 , c i 1 , c i 2 of token w

Original: i are [ w i ] , [ w i − 1 ; w i ] and [ w i ; w i +1 ] respectively.
Revised: i are [ w i ] , [ w i − 1 ; w i ] and [ w i ; w i +1 ] respectively.

Original: Then we encode each context c ij to a vector z c ij by feeding pre-trained word embeddings into a bi-directional LSTM encoder (Hochreiter & Schmidhuber, 1997).
Revised: Then we encode each context c ij to a vector z c ij by feeding pre-trained word embeddings into a bi-directional LSTM encoder (Hochreiter & Schmidhuber, 1997).

Original: All hidden layers of BiLSTM are then summarized by an attention layer (Bahdanau et al., 2014).
Revised: All hidden layers of BiLSTM are then summarized by an attention layer (Bahdanau et al., 2014).

Original: For keyword query q , we directly encode it into vector z q by bi-LSTM and attention.
Revised: For keyword query q , we directly encode it into vector z q by bi-LSTM and attention.

Original: Finally, scores of sentence x and query q are calculated by aggregating similarity scores from different sliding windows:
Revised: Finally, scores of sentence x and query q are calculated by aggregating similarity scores from different sliding windows:

Original: [Equation]
Revised: [Equation]

Original: where D is a trainable diagonal matrix, v ∈ R N c is the trainable weight of each sliding window.
Revised: where D is a trainable diagonal matrix, v ∈ R N c is the trainable weight of each sliding window.

Original: Parameters in the string matching module need to be learned with data in the form of (sentence, keyword, label) .
Revised: Parameters in the string matching module need to be learned with data in the form of (sentence, keyword, label) .

Original: To build a training set for learning string matching, we randomly select spans of consecutive words as keyword queries in the training data.
Revised: To build a training set for learning string matching, we randomly select spans of consecutive words as keyword queries in the training data.

Original: Each query is paired with either the sentence it comes from or a different sentence.
Revised: Each query is paired with either the sentence it comes from or a different sentence.

Original: The synthesized dataset is denoted as { x i , q i , k i } N syn i =1 , where k ij will take the value of 1 if q is extracted from x ij and 0 otherwise.
Revised: The synthesized dataset is denoted as { x i , q i , k i } N syn i =1 , where k ij will take the value of 1 if q is extracted from x ij and 0 otherwise.

Original: The loss function is deﬁned as the binary cross-entropy loss, as follows.
Revised: The loss function is deﬁned as the binary cross-entropy loss, as follows.

Original: [Equation]
Revised: [Equation]

Original: While pretraining with L find enables matching similar words, this unsupervised distributional method is poor at learning their semantic meanings.
Revised: While pretraining with L find enables matching similar words, this unsupervised distributional method is poor at learning their semantic meanings.

Original: For example, the word “good” will have very high similarity to “bad” because they often coexist with the same set of context words.
Revised: For example, the word “good” will have very high similarity to “bad” because they often coexist with the same set of context words.

Original: To solve this problem, we borrow the idea of word retroﬁtting (Faruqui et al., 2014) and adopt a contrastive loss (Neculoiu et al., 2016) to incorporate semantic knowledge in training.
Revised: To solve this problem, we borrow the idea of word retroﬁtting (Faruqui et al., 2014) and adopt a contrastive loss (Neculoiu et al., 2016) to incorporate semantic knowledge in training.

Original: We use the keyword queries in labeling functions as supervision.
Revised: We use the keyword queries in labeling functions as supervision.

Original: Intuitively, the semantic meaning of two queries should be similar if they appear in the same class of labeling functions and dissimilar otherwise.
Revised: Intuitively, the semantic meaning of two queries should be similar if they appear in the same class of labeling functions and dissimilar otherwise.

Original: More specifically, for a query q , we denote queries in the same class of labeling functions as Q + and queries in different classes of labeling functions as Q − .
Revised: More specifically, for a query q , we denote queries in the same class of labeling functions as Q + and queries in different classes of labeling functions as Q − .

Original: The similarity loss is deﬁned as:
Revised: The similarity loss is deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: The overall objective function for string matching module is:
Revised: The overall objective function for string matching module is:

Original: [Equation]
Revised: [Equation]

Original: where γ is a hyper-parameter.
Revised: where γ is a hyper-parameter.

Original: We pretrain the string matching module for better initialization.
Revised: We pretrain the string matching module for better initialization.

Original: Soft Counting Module.
Revised: Soft Counting Module.

Original: The soft counting module aims to relax the counting (distance) constraints deﬁned by NL explanations.
Revised: The soft counting module aims to relax the counting (distance) constraints deﬁned by NL explanations.

Original: For a counting constraint precede object by no more than three words , the soft counting module outputs a matching score indicating to what extent an anchor word (TERM,[Table] [Table] [Table]
Revised: For a counting constraint precede object by no more than three words , the soft counting module outputs a matching score indicating to what extent an anchor word (TERM,

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]
Revised: [Table]

Original: SUBJECT, and OBJECT) satisﬁes the constraint.
Revised: SUBJECT, and OBJECT) satisﬁes the constraint.

Original: The score is set to 1 if the position of the anchor word strictly satisﬁes the constraint, and will decrease if the constraint is broken.
Revised: The score is set to 1 if the position of the anchor word strictly satisﬁes the constraint, and will decrease if the constraint is broken.

Original: For simplicity, we allow an additional range in which the score is set to µ ∈ (0 , 1) , which is a hyper-parameter controlling the constraints.
Revised: For simplicity, we allow an additional range in which the score is set to µ ∈ (0 , 1) , which is a hyper-parameter controlling the constraints.

Original: Deterministic Function Module.
Revised: Deterministic Function Module.

Original: The deterministic function module deals with deterministic predicates like “Left” and “Right”, which can only be exactly matched because a string is either right or left of an anchor word.
Revised: The deterministic function module deals with deterministic predicates like “Left” and “Right”, which can only be exactly matched because a string is either right or left of an anchor word.

Original: Therefore, the probability it outputs should be either 0 or 1.
Revised: Therefore, the probability it outputs should be either 0 or 1.

Original: The Deterministic Function Module deals with all these predicates and outputs a mask sequence, which is fed into the tree structure and combined with other information.
Revised: The Deterministic Function Module deals with all these predicates and outputs a mask sequence, which is fed into the tree structure and combined with other information.

Original: Logical Calculation Module.
Revised: Logical Calculation Module.

Original: The logical calculation module acts as a score aggregator.
Revised: The logical calculation module acts as a score aggregator.

Original: It can aggregate scores given by: (1) a string matching module and a soft counting module/deterministic function module (triggered by predicates such as “Is” and “Occur”) (2) two clauses that have been evaluated with a score respectively (triggered by predicates such as “And” and “Or”).
Revised: It can aggregate scores given by: (1) a string matching module and a soft counting module/deterministic function module (triggered by predicates such as “Is” and “Occur”) (2) two clauses that have been evaluated with a score respectively (triggered by predicates such as “And” and “Or”).

Original: In the ﬁrst case, the logical calculation module will calculate the element-wise products of the score sequence provided by the string matching module and the mask sequence provided by the soft counting module/deterministic function module.
Revised: In the ﬁrst case, the logical calculation module will calculate the element-wise products of the score sequence provided by the string matching module and the mask sequence provided by the soft counting module/deterministic function module.

Original: It then uses max pooling to calculate the matching score of the current clause.
Revised: It then uses max pooling to calculate the matching score of the current clause.

Original: In the second case, the logical calculation module will aggregate the scores of at least one clause based on the logic operation.
Revised: In the second case, the logical calculation module will aggregate the scores of at least one clause based on the logic operation.

Original: The rules are deﬁned as follows.
Revised: The rules are deﬁned as follows.

Original: [Equation]
Revised: [Equation]

Original: where p is the score of the input clause.
Revised: where p is the score of the input clause.

Original: 3. A UGMENTING M ODEL L EARNING WITH NMET
Revised: 3. A UGMENTING M ODEL L EARNING WITH NE X T

Original: As described in Algo.
Revised: As described in Algo.

Original: 1, in each iteration, we sample two batches B a and B u from S a and S u .
Revised: 1, in each iteration, we sample two batches B a and B u from S a and S u .

Original: We conduct supervised learning on B a .
Revised: We conduct supervised learning on B a .

Original: The labeled loss function is calculated as:
Revised: The labeled loss function is calculated as:

Original: [Equation]
Revised: [Equation]

Original: To leverage B u , which is also informative, for each instance x uj ∈ B u , we can use our matching modules to compute its matching score with every logical form.
Revised: To leverage B u , which is also informative, for each instance x uj ∈ B u , we can use our matching modules to compute its matching score with every logical form.

Original: The most probable logical form that matched with x uj is denoted as y uj 1 , along with the matching score u j .
Revised: The most probable logical form that matched with x uj is denoted as y uj 1 , along with the matching score u j .

Original: To ensure the scale of the unlabeled loss is comparable to labeled loss, we normalize the matching scores among pseudolabeled instances in B u as:
Revised: To ensure the scale of the unlabeled loss is comparable to labeled loss, we normalize the matching scores among pseudolabeled instances in B u as:

Original: [Equation]
Revised: [Equation]

Original: where k is the index of the instance and hyperparameter θ t (temperature) controls the shape of normalized scores’ distribution.
Revised: where k is the index of the instance and hyperparameter θ t (temperature) controls the shape of normalized scores’ distribution.

Original: Based on that, the unlabeled loss is calculated as:
Revised: Based on that, the unlabeled loss is calculated as:

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: Note that the string matching module is also trainable and plays a vital role in NMET. We jointly learn it with the classiﬁer by optimizing:
Revised: Note that the string matching module is also trainable and plays a vital role in NExT. We jointly learn it with the classiﬁer by optimizing:

Original: [Equation]
Revised: [Equation]

Original: where α and β are hyper-parameters.
Revised: where α and β are hyper-parameters.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: Tasks and Datasets.
Revised: Tasks and Datasets.

Original: We conduct experiments on two tasks: relation extraction and aspect-termlevel sentiment analysis.
Revised: We conduct experiments on two tasks: relation extraction and aspect-termlevel sentiment analysis.

Original: Relation extraction (RE) aims to identify the relation type between two entities in a sentence.
Revised: Relation extraction (RE) aims to identify the relation type between two entities in a sentence.

Original: For example, given a sentence Steve Jobs founded Apple Inc , we want to extract a triple ( Steve Jobs , Apple Inc. , Founder ).
Revised: For example, given a sentence Steve Jobs founded Apple Inc , we want to extract a triple ( Steve Jobs , Apple Inc. , Founder ).

Original: For RE we choose two datasets, TACRED (Zhang et al., 2017) and SemEval (Hendrickx et al., 2009) in our experiments.
Revised: For RE we choose two datasets, TACRED (Zhang et al., 2017) and SemEval (Hendrickx et al., 2009) in our experiments.

Original: Aspect-term-level sentiment analysis (SA) aims to decide the sentiment polarity with regard to the given aspect term.
Revised: Aspect-term-level sentiment analysis (SA) aims to decide the sentiment polarity with regard to the given aspect term.

Original: For example, given a sentence Quality ingredients preparation all around, and a very fair price for NYC , the sentiment polarity of the aspect term price is positive, the explanation can be The word price is directly preceded by fair .
Revised: For example, given a sentence Quality ingredients preparation all around, and a very fair price for NYC , the sentiment polarity of the aspect term price is positive, the explanation can be The word price is directly preceded by fair .

Original: For this task we use two customer reviews datasets, Restaurant and
Revised: For this task we use two customer reviews datasets, Restaurant and

Original: Laptop, which are part of SemEval 

Revised: Laptop, which are part of SemEval

Original: Task 4.
Revised: Task 4.

Original: Explanation Collection.
Revised: Explanation Collection.

Original: We use Amazon Mechanical Turk to collect labels and explanations for a randomly sampled set of instances in each dataset.
Revised: We use Amazon Mechanical Turk to collect labels and explanations for a randomly sampled set of instances in each dataset.

Original: Turkers are prompted with a list of selected predicates (see Appendix) and several examples of NL explanation.
Revised: Turkers are prompted with a list of selected predicates (see Appendix) and several examples of NL explanation.

Original: Examples of collected explanations are listed in Appendix.
Revised: Examples of collected explanations are listed in Appendix.

Original: Statistics of curated explanations and intrinsic evaluation results of semantic parsing are summarized in Table 1.
Revised: Statistics of curated explanations and intrinsic evaluation results of semantic parsing are summarized in Table 1.

Original: To ensure a low-resource setting (i.e., |S (cid:48) | (cid:28) |S| ), in each experiment we only use a random subset of collected explanations.
Revised: To ensure a low-resource setting (i.e., |S (cid:48) | (cid:28) |S| ), in each experiment we only use a random subset of collected explanations.

Original: Compared Methods.
Revised: Compared Methods.

Original: As mentioned in Sec. 2, logic forms partitioned unlabeled corpus S into labeled set S a and unlabeled set S u .
Revised: As mentioned in Sec. 2, logic forms partitioned unlabeled corpus S into labeled set S a and unlabeled set S u .

Original: Labeled set S a can be directly utilized by supervised learning methods.
Revised: Labeled set S a can be directly utilized by supervised learning methods.

Original: (1) CBOW-GloVe uses bag-of-words (Mikolov et al., 2013) on GloVe embeddings (Pennington et al., 2014) to represent an instance, or surface patterns in NL explanation.
Revised: (1) CBOW-GloVe uses bag-of-words (Mikolov et al., 2013) on GloVe embeddings (Pennington et al., 2014) to represent an instance, or surface patterns in NL explanation.

Original: It then annotates the sentence with the label of it most similar surface pattern (as with cosine similarity).
Revised: It then annotates the sentence with the label of it most similar surface pattern (as with cosine similarity).

Original: PCNN (Zeng et al., 2015) uses piece-wise max-pooling to aggregate CNN-generated features.
Revised: PCNN (Zeng et al., 2015) uses piece-wise max-pooling to aggregate CNN-generated features.

Original: LSTM+ATT (Bahdanau et al., 2014) adds an attention layer onto LSTM to encode an sequence.
Revised: LSTM+ATT (Bahdanau et al., 2014) adds an attention layer onto LSTM to encode an sequence.

Original: PA-LSTM (Zhang et al., 2017) combines LSTM with an entity-position aware attention to conduct relation extraction.
Revised: PA-LSTM (Zhang et al., 2017) combines LSTM with an entity-position aware attention to conduct relation extraction.

Original: (5) ATAE-LSTM (Wang et al., 2016) combines the aspect term information into both embedding layer and attention layer to help the model concentrate on different parts of a sentence.
Revised: (5) ATAE-LSTM (Wang et al., 2016) combines the aspect term information into both embedding layer and attention layer to help the model concentrate on different parts of a sentence.

Original: In semi-supervised baselines, unlabeled data S u are also introduced to training.
Revised: In semi-supervised baselines, unlabeled data S u are also introduced to training.

Original: For methods requiring rules as input, we use surface pattern-based rules transferred from explanations.
Revised: For methods requiring rules as input, we use surface pattern-based rules transferred from explanations.

Original: Compared semi-supervised methods include: (1) Pseudo-Labeling (Lee, 2013)
Revised: Compared semi-supervised methods include: (1) Pseudo-Labeling (Lee, 2013)

Original: ﬁrst trains a classiﬁer on labeled dataset, then generate pseudo labels for unlabeled data using the classiﬁer by selecting the label with maximum predicted probability.
Revised: ﬁrst trains a classiﬁer on labeled dataset, then generate pseudo labels for unlabeled data using the classiﬁer by selecting the label with maximum predicted probability.

Original: (2) Self-Training (Rosenberg et al., 2005) proposes to expand the labeled data by selecting a batch of unlabeled data that has the highest conﬁdence and generate pseudo-labels for them.
Revised: (2) Self-Training (Rosenberg et al., 2005) proposes to expand the labeled data by selecting a batch of unlabeled data that has the highest conﬁdence and generate pseudo-labels for them.

Original: The method stop until the unlabeled data is used up.
Revised: The method stop until the unlabeled data is used up.

Original: Mean-Teacher (Tarvainen & Valpola, 2017) averages model weights instead of label predictions and assumes similar data points should have similar outputs.
Revised: Mean-Teacher (Tarvainen & Valpola, 2017) averages model weights instead of label predictions and assumes similar data points should have similar outputs.

Original: (4) DualRE (Lin et al., 2019) jointly trains a relation prediction module and a retrieval module.
Revised: (4) DualRE (Lin et al., 2019) jointly trains a relation prediction module and a retrieval module.

Original: Learning from explanations is categorized as a third setting.
Revised: Learning from explanations is categorized as a third setting.

Original: Both methods generate explanationguided pseudo labels for a downstream classiﬁer.
Revised: Both methods generate explanationguided pseudo labels for a downstream classiﬁer.

Original: (1) Data Programming (Hancock et al., 2018;
Revised: (1) Data Programming (Hancock et al., 2018;

Original: Ratner et al., 2016) aggregates results of strict labeling functions for each instance and use these pseudo-labels to train a classiﬁer.
Revised: Ratner et al., 2016) aggregates results of strict labeling functions for each instance and use these pseudo-labels to train a classiﬁer.

Original: (2) NMET (proposed work) softly applies logic forms to get annotations for unlabeled instances and train a downstream classiﬁer with these pseudo-labeled instances.
Revised: (2) NExT (proposed work) softly applies logic forms to get annotations for unlabeled instances and train a downstream classiﬁer with these pseudo-labeled instances.

Original: The downstream classiﬁer is BiLSTM+ATT for relation extraction and ATAE-LSTM for sentiment analysis.
Revised: The downstream classiﬁer is BiLSTM+ATT for relation extraction and ATAE-LSTM for sentiment analysis.

Original: 4. R ESULTS O VERVIEW
Revised: 4. R ESULTS O VERVIEW

Original: Table 2 (a) lists F1 scores of all relation extraction models.
Revised: Table 2 (a) lists F1 scores of all relation extraction models.

Original: Full results including precision and recall can be found in Appendix A.4.
Revised: Full results including precision and recall can be found in Appendix A.4.

Original: We observe that our proposed NMET consistently outperform all baseline models in low-resource setting.
Revised: We observe that our proposed NExT consistently outperform all baseline models in low-resource setting.

Original: Also, we found that, (1) Directly applying logic forms to unlabeled data results in poor performance.
Revised: Also, we found that, (1) Directly applying logic forms to unlabeled data results in poor performance.

Original: We notice that this method achieves high precision but low recall.
Revised: We notice that this method achieves high precision but low recall.

Original: Based on our observation of the collected dataset, this is because people tend to use detailed and speciﬁc constraints in an NL explanation to ensure they cover all aspects of the instance.
Revised: Based on our observation of the collected dataset, this is because people tend to use detailed and speciﬁc constraints in an NL explanation to ensure they cover all aspects of the instance.

Original: As a result, those instances that satisfy the constraints are correctly labeled in most cases, and thus the precision is high.
Revised: As a result, those instances that satisfy the constraints are correctly labeled in most cases, and thus the precision is high.

Original: Meanwhile, generalization ability is compromised, and thus the recall is low.
Revised: Meanwhile, generalization ability is compromised, and thus the recall is low.

Original: (2) Compared to its downstream classiﬁer baseline (BiLSTM+ATT with S a ), NMET achieves 4.2% F1 improvement in absolute value on TACRED, and 5.5% on SemEval.
Revised: Compared to its downstream classiﬁer baseline (BiLSTM+ATT with S a ), NExT achieves 4.2% F1 improvement in absolute value on TACRED, and 5.5% on SemEval.

Original: This validates that the expansion of rule coverage by NMET is effective and is providing useful information to classiﬁer training.
Revised: This validates that the expansion of rule coverage by NExT is effective and is providing useful information to classiﬁer training.

Original: (3) Performance gap further widens when we take annotation efforts into account.
Revised: (3) Performance gap further widens when we take annotation efforts into account.

Original: The annotation time for E and S l are equivalent; but the performance of BiLSTM+ATT signiﬁcantly degrades with fewer instances in S l .
Revised: The annotation time for E and S l are equivalent; but the performance of BiLSTM+ATT signiﬁcantly degrades with fewer instances in S l .

Original: (4) Results of semi-supervised methods are unsatisfactory.
Revised: (4) Results of semi-supervised methods are unsatisfactory.

Original: This may be explained with difference between underlying data distribution of S a and S u .
Revised: This may be explained with difference between underlying data distribution of S a and S u .

Original: Table 2 (b) lists the performances of all sentiment analysis models.
Revised: Table 2 (b) lists the performances of all sentiment analysis models.

Original: The observations are similar to those of relation extraction, which strengthen our conclusions and validates the capability of NMET.
Revised: The observations are similar to those of relation extraction, which strengthen our conclusions and validates the capability of NExT

Original: [Table]
Revised: [Table]

Original: 4. P ERFORMANCE A NALYSIS
Revised: 4. P ERFORMANCE A NALYSIS

Original: Effectiveness of softening logical rules.
Revised: Effectiveness of softening logical rules.

Original: As shown in Table
Revised: As shown in Table

Original: [Table]
Revised: [Table]

Original: we conduct ablation studies on TACRED and Restaurant.
Revised: we conduct ablation studies on TACRED and Restaurant.

Original: We remove two modules that support soft logic (by only allowing them to give 0/1 outputs) to see how much does rule softening help in our framework.
Revised: We remove two modules that support soft logic (by only allowing them to give 0/1 outputs) to see how much does rule softening help in our framework.

Original: Both soft counting module and string matching module contribute to the performance of NMET. It can be easily concluded that string matching module plays a vital role.
Revised: Both soft counting module and string matching module contribute to the performance of NExT. It can be easily concluded that string matching module plays a vital role.

Original: Removing it leads to signiﬁcant performance drops, which demonstrates the effectiveness of generalizing when applying logical forms.
Revised: Removing it leads to signiﬁcant performance drops, which demonstrates the effectiveness of generalizing when applying logical forms.

Original: Besides, we examine the impact brought by L sim , L find .
Revised: Besides, we examine the impact brought by L sim , L find .

Original: Removing them severely hurts the performance, indicating the importance of semantic learning when performing fuzzy matching.
Revised: Removing them severely hurts the performance, indicating the importance of semantic learning when performing fuzzy matching.

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: Superiority of explanations in data efﬁciency.
Revised: Superiority of explanations in data efﬁciency.

Original: In the real world, with limited human-power, there is a question of whether it is more valuable to spend time explaining existing annotations than just annotating more labels.
Revised: In the real world, with limited human-power, there is a question of whether it is more valuable to spend time explaining existing annotations than just annotating more labels.

Original: To answer this question, we carefully examine the data efﬁciency of explanations over pure labels.
Revised: To answer this question, we carefully examine the data efﬁciency of explanations over pure labels.

Original: For skilled annotators, the time for labeling one instance in TACRED plus giving an explanation is as 2 times as the time for only labeling one instance.
Revised: For skilled annotators, the time for labeling one instance in TACRED plus giving an explanation is as 2 times as the time for only labeling one instance.

Original: For Restaurant, the ratio is 2.3.
Revised: For Restaurant, the ratio is 2.3.

Original: We then compare the performance between these two labeling strategies within equal annotation time, e.g. 100 explanations v.s. 200 labeled data in TACRED.
Revised: We then compare the performance between these two labeling strategies within equal annotation time, e.g. 100 explanations v.s. 200 labeled data in TACRED.

Original: From Fig.
Revised: From Fig.

Original: 5 (a) and (b), we can see that our NMET beats purely-supervised and semi-supervised baselines in most cases, even though they are given more labels.
Revised: 5 (a) and (b), we can see that our NExT beats purely-supervised and semi-supervised baselines in most cases, even though they are given more labels.

Original: We argue that giving explanations surely yields additional cost, but logical form based matching has high accuracy and can generalize well with our NMET.
Revised: We argue that giving explanations surely yields additional cost, but logical form based matching has high accuracy and can generalize well with our NExT.

Original: Explanations prove to be a very efﬁcient form for data annotation.
Revised: Explanations prove to be a very efﬁcient form for data annotation.

Original: Performance with different number of explanations.
Revised: Performance with different number of explanations.

Original: From Fig.
Revised: From Fig.

Original: 5 (c) and (d), one can clearly observe that all approaches beneﬁt from more labeled data.
Revised: 5 (c) and (d), one can clearly observe that all approaches beneﬁt from more labeled data.

Original: Our NMET outperforms all other baselines by a large margin, which indicates the effectiveness of leveraging knowledge embedded in NL explanations.
Revised: Our NExT outperforms all other baselines by a large margin, which indicates the effectiveness of leveraging knowledge embedded in NL explanations.

Original: Performance with different amount of unlabeled data.
Revised: Performance with different amount of unlabeled data.

Original: To investigate how our NMET’s performance is affected by the amount of unlabeled data, we randomly sample 10% , 30% , 50% and 70% of the original unlabeled dataset to do the experiments.
Revised: To investigate how our NExT’s performance is affected by the amount of unlabeled data, we randomly sample 10% , 30% , 50% and 70% of the original unlabeled dataset to do the experiments.

Original: As illustrated in Fig.
Revised: As illustrated in Fig.

Original: 3, our NMET beneﬁts from larger amount of unlabeled data.
Revised: 3, our NExT beneﬁts from larger amount of unlabeled data.

Original: We attribute it to high accuracy of logical forms converted from explanations.
Revised: We attribute it to high accuracy of logical forms converted from explanations.

Original: Case study on string matching module.
Revised: Case study on string matching module.

Original: String matching module plays a vital role in NMET. The matching quality greatly inﬂuences the accuracy of pseudo labeling.
Revised: String matching module plays a vital role in NExT. The matching quality greatly inﬂuences the accuracy of pseudo labeling.

Original: In Fig.
Revised: In Fig.

Original: 4, we can see that keyword chief executive of is perfectly aligned with executive director of in the sentence, which demonstrates the effectiveness of string matching module in capturing semantic similarity.
Revised: 4, we can see that keyword chief executive of is perfectly aligned with executive director of in the sentence, which demonstrates the effectiveness of string matching module in capturing semantic similarity.

Original: 4. A DDITIONAL E XPERIMENT ON M ULTI - HOP R EASONING
Revised: 4. A DDITIONAL E XPERIMENT ON M ULTI - HOP R EASONING

Original: To further test the capability of NMET in downstream tasks, we apply it to W IKI H OP
Revised: To further test the capability of NExT in downstream tasks, we apply it to W IKI H OP

Original: (Welbl et al., 2018) country task by fusing NMET-matched facts into baseline model NLP ROLOG (Weber et al.,
Revised: (Welbl et al., 2018) country task by fusing NExT-matched facts into baseline model NLP ROLOG (Weber et al.,

Original: [Table]
Revised: [Table]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: For a brief introduction, W IKI H OP country task requires a model to select the correct candidate ENT-Y for question “Which country is ENT-X in?” given a list of support sentences.
Revised: For a brief introduction, W IKI H OP country task requires a model to select the correct candidate ENT-Y for question “Which country is ENT-X in?” given a list of support sentences.

Original: As part of dataset design, the correct answer can only be found by reasoning over multiple support sentences
Revised: As part of dataset design, the correct answer can only be found by reasoning over multiple support sentences

Original: [Figure]
Revised: [Figure]

Original: NLP ROLOG is a model proposed for W IKI H OP .
Revised: NLP ROLOG is a model proposed for W IKI H OP .

Original: It ﬁrst extract triples from support sentences and treat the masked sentence as the relation between two entities.
Revised: It ﬁrst extract triples from support sentences and treat the masked sentence as the relation between two entities.

Original: For example, “Socrate was born in
Revised: For example, “Socrate was born in

Original: Athens” is converted to (Socrate, “ ENT1 was born in ENT2 ”, Athens), where “ ENT1 was born in
Revised: Athens” is converted to (Socrate, “ ENT1 was born in ENT2 ”, Athens), where “ ENT1 was born in

Original: ENT2 ” is later embedded by S ENT 2V EC (Pagliardini et al., 2018) to represent the relation between
Revised: ENT2 ” is later embedded by S ENT 2V EC (Pagliardini et al., 2018) to represent the relation between

Original: ENT1 and ENT2 .
Revised: ENT1 and ENT2 .

Original: Triples extracted from supporting sentences are fed into a Prolog reasoner which will do backward chaining and reasoning to arrive at the target statement country( ENT-X , ENT-Y ).
Revised: Triples extracted from supporting sentences are fed into a Prolog reasoner which will do backward chaining and reasoning to arrive at the target statement country( ENT-X , ENT-Y ).

Original: We refer readers to (Weber et al., 2019) for in-depth introduction of NLProlog NLP ROLOG .
Revised: We refer readers to (Weber et al., 2019) for in-depth introduction of NLProlog NLP ROLOG .

Original: Fig.
Revised: Fig.

Original: 6 shows how the framework in Fig.
Revised: 6 shows how the framework in Fig.

Original: 1 is adjusted to suit NLP ROLOG .
Revised: 1 is adjusted to suit NLP ROLOG .

Original: We manually choose 3 predicates (i.e., located in, capital of, next to) and annotate 21 support sentences with natural language explanation.
Revised: We manually choose 3 predicates (i.e., located in, capital of, next to) and annotate 21 support sentences with natural language explanation.

Original: We get 103 strictly-matched facts ( S a ) and 1407 NMET-matched facts ( S u ) among the 128k unlabeled QA support sentences.
Revised: We get 103 strictly-matched facts ( S a ) and 1407 NExT-matched facts ( S u ) among the 128k unlabeled QA support sentences.

Original: Additionally, we manually write 5 rules about these 3 predicates for the Prolog solver, e.g. located in(X,Z) ← located in(X,Y) ∧ located in(Y,Z).
Revised: Additionally, we manually write 5 rules about these 3 predicates for the Prolog solver, e.g. located in(X,Z) ← located in(X,Y) ∧ located in(Y,Z).

Original: Results are listed in Table 4.
Revised: Results are listed in Table 4.

Original: From the result we observe that simply adding the 103 strictly-matched facts is not making notable improvement.
Revised: From the result we observe that simply adding the 103 strictly-matched facts is not making notable improvement.

Original: However, with the help of NMET, a larger number of structured facts are recognized from support sentences, so that external knowledge from onlyexplanations and 5 rules improve the accuracy by 1 point.
Revised: However, with the help of NExT, a larger number of structured facts are recognized from support sentences, so that external knowledge from only 21 explanations and 5 rules improve the accuracy by 1 point.

Original: This observation validates NMET’s capability in low resource setting and highlight its potential when applied to downstream tasks.
Revised: This observation validates NExT’s capability in low resource setting and highlight its potential when applied to downstream tasks.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Leveraging natural language for training classiﬁers.
Revised: Leveraging natural language for training classiﬁers.

Original: Supervision in the form of natural language has been explored by many works.
Revised: Supervision in the form of natural language has been explored by many works.

Original: Srivastava et al.
Revised: Srivastava et al.

Original: ﬁrst demonstrates the effectiveness of NL explanations.
Revised: ﬁrst demonstrates the effectiveness of NL explanations.

Original: They proposed a joint concept learning and semantic parsing method for classiﬁcation problems.
Revised: They proposed a joint concept learning and semantic parsing method for classiﬁcation problems.

Original: However, the method is very limited in that it is not able to use unlabeled data.
Revised: However, the method is very limited in that it is not able to use unlabeled data.

Original: To address this issue, Hancock et al.
Revised: To address this issue, Hancock et al.

Original: (2018) propose to parse the NL explanations into labeling functions
Revised: (2018) propose to parse the NL explanations into labeling functions

Original: [Figure]
Revised: [Figure]

Original: Raw Corpus + NL Explanations
Revised: Raw Corpus + NL Explanations

Original: Semantic

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Table] [Figure] [Table]
Revised: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Table]
Revised: [Table]

Original: and then use data programming to handle the conﬂict and enhancement between different labeling functions.
Revised: and then use data programming to handle the conﬂict and enhancement between different labeling functions.

Original: Camburu et al.
Revised: Camburu et al.

Original: (2018) extend Stanford Natural Language Inference dataset with NL explanations and demonstrate its usefulness for various goals for training classiﬁers.
Revised: (2018) extend Stanford Natural Language Inference dataset with NL explanations and demonstrate its usefulness for various goals for training classiﬁers.

Original: Andreas et al.
Revised: Andreas et al.

Original: (2016) explores decomposing NL questions into linguistic substructures for learning collections of neural modules which can be assembled into deep networks.
Revised: (2016) explores decomposing NL questions into linguistic substructures for learning collections of neural modules which can be assembled into deep networks.

Original: Hu et al.
Revised: Hu et al.

Original: (2019) explores using
Revised: (2019) explores using

Original: NL instructions as compositional representation of actions for hierarchical decision making.
Revised: NL instructions as compositional representation of actions for hierarchical decision making.

Original: The substructure of an instruction is summarized as a latent plan, which is then executed by another model.
Revised: The substructure of an instruction is summarized as a latent plan, which is then executed by another model.

Original: Rajani et al.
Revised: Rajani et al.

Original: (2019) train a language model to automatically generate NL explanations that can be used during training and inference for the task of commonsense reasoning.
Revised: (2019) train a language model to automatically generate NL explanations that can be used during training and inference for the task of commonsense reasoning.

Original: Weakly-supervised learning.
Revised: Weakly-supervised learning.

Original: Our work is relevant to weakly-supervised learning.
Revised: Our work is relevant to weakly-supervised learning.

Original: Traditional systems use handcrafted rules (Hearst, 1992) or automatically learned rules (Agichtein & Gravano, 2000; Batista et al., 2015) to take a rule-based approach.
Revised: Traditional systems use handcrafted rules (Hearst, 1992) or automatically learned rules (Agichtein & Gravano, 2000; Batista et al., 2015) to take a rule-based approach.

Original: Hu et al.
Revised: Hu et al.

Original: (2019) incorporate human knowledge into neural networks by using a teacher network to teach the classiﬁer knowledge from rules and train the classiﬁer with labeled data.
Revised: (2019) incorporate human knowledge into neural networks by using a teacher network to teach the classiﬁer knowledge from rules and train the classiﬁer with labeled data.

Original: Li et al.
Revised: Li et al.

Original: (2018) parse regular expression to get action trees as a classiﬁer that are composed of neural modules, so that essentially training stage is just a process of learning human knowledge.
Revised: (2018) parse regular expression to get action trees as a classiﬁer that are composed of neural modules, so that essentially training stage is just a process of learning human knowledge.

Original: Meanwhile, if we regard those data that are exactly matched by rules as labeled data and the remaining as unlabeled data, we can apply many semi-supervised models such as self learning (Rosenberg et al., 2005), mean-teacher (Tarvainen & Valpola, 2017), and semi-supervised VAE (Xu et al., 2017).
Revised: Meanwhile, if we regard those data that are exactly matched by rules as labeled data and the remaining as unlabeled data, we can apply many semi-supervised models such as self learning (Rosenberg et al., 2005), mean-teacher (Tarvainen & Valpola, 2017), and semi-supervised VAE (Xu et al., 2017).

Original: However, These models turn out to be ineffective in rule-labeled data or explanation-labeled data due to potentially large difference in label distribution.
Revised: However, These models turn out to be ineffective in rule-labeled data or explanation-labeled data due to potentially large difference in label distribution.

Original: The data sparsity is also partially solved by distant supervision (Mintz et al., 2009; Surdeanu et al., 2012).
Revised: The data sparsity is also partially solved by distant supervision (Mintz et al., 2009; Surdeanu et al., 2012).

Original: They rely on knowledge bases (KBs) to annotate data.
Revised: They rely on knowledge bases (KBs) to annotate data.

Original: However, the methods introduce a lot of noise, which severely hinders the performance.
Revised: However, the methods introduce a lot of noise, which severely hinders the performance.

Original: Liu et al.
Revised: Liu et al.

Original: (2017) instead propose to conduct relation extraction using annotations from heterogeneous information source.
Revised: (2017) instead propose to conduct relation extraction using annotations from heterogeneous information source.

Original: Again, predicting true labels from noisy sources is challenging.
Revised: Again, predicting true labels from noisy sources is challenging.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we presented NMET, a framework that augments sequence classiﬁcation by exploiting
Revised: In this paper, we presented NExT, a framework that augments sequence classiﬁcation by exploiting

Original: NL explanations as supervision under a low resource setting.
Revised: NL explanations as supervision under a low resource setting.

Original: We tackled the challenges of modeling the compositionality of NL explanations and dealing with the linguistic variants.
Revised: We tackled the challenges of modeling the compositionality of NL explanations and dealing with the linguistic variants.

Original: Four types of modules were introduced to generalize the different types of actions in logical forms, which substantially increases the coverage of NL explanations.
Revised: Four types of modules were introduced to generalize the different types of actions in logical forms, which substantially increases the coverage of NL explanations.

Original: A joint training algorithm was proposed to utilize information from both labeled dataset and unlabeled dataset.
Revised: A joint training algorithm was proposed to utilize information from both labeled dataset and unlabeled dataset.

Original: We conducted extensive experiments on several datasets and proved the effectiveness of our model.
Revised: We conducted extensive experiments on several datasets and proved the effectiveness of our model.

Original: Future work includes extending NMET to sequence labeling tasks and building a cross-domain semantic parser for NL explanations.
Revised: Future work includes extending NExT to sequence labeling tasks and building a cross-domain semantic parser for NL explanations.

Original: A A PPENDIX A. P REDICATES
Revised: A A PPENDIX A. P REDICATES

Original: Following Srivastava et al.
Revised: Following Srivastava et al.

Original: (2017), we ﬁrst compile a domain lexicon that maps each word to its syntax and logical predicate.
Revised: (2017), we ﬁrst compile a domain lexicon that maps each word to its syntax and logical predicate.

Original: Table 5 lists some frequently used predicates in our parser, descriptions about their function and modules they belong to.
Revised: Table 5 lists some frequently used predicates in our parser, descriptions about their function and modules they belong to.

Original: Predicate
Revised: Predicate

Original: [Table]
Revised: [Table]

Original: Separator
Revised: Separator

Original: [Table]
Revised: [Table]

Original: A. E XAMPLES FOR COLLECTED EXPLANATIONS . TACRED
Revised: A. E XAMPLES FOR COLLECTED EXPLANATIONS . TACRED

Original: Although not a Playboy Playmate , she has appeared in nude pictorials with her Girls Next Door costars and fellow Hefner girlfriends Holly Madison and OBJ-PERSON , then known as SUBJPERSON.
Revised: Although not a Playboy Playmate , she has appeared in nude pictorials with her Girls Next Door costars and fellow Hefner girlfriends Holly Madison and OBJ-PERSON , then known as SUBJPERSON.

Original: (Label) per:alternate names (Explanation) the term ‘‘then known as’’ occurs between
Revised: (Label) per:alternate names (Explanation) the term ‘‘then known as’’ occurs between

Original: SUBJ-PERSON and OBJ-PERSON
Revised: SUBJ-PERSON and OBJ-PERSON

Original: and there are no more than six words between SUBJ-PERSON and OBJ-PERSON.
Revised: and there are no more than six words between SUBJ-PERSON and OBJ-PERSON.

Original: Ofﬁcials in Mumbai said that the two suspects , David Coleman Headley , an American with links of Pakistan , and SUBJ-PERSON , who was born in Pakistan but is a OBJ-NATIONALITY citizen , both visited Mumbai and several other Indian cities in before the attacks , and may have visited some of the sites that were attacked.
Revised: Ofﬁcials in Mumbai said that the two suspects , David Coleman Headley , an American with links of Pakistan , and SUBJ-PERSON , who was born in Pakistan but is a OBJ-NATIONALITY citizen , both visited Mumbai and several other Indian cities in before the attacks , and may have visited some of the sites that were attacked.

Original: (Label) per:origin (Explanation the words ‘‘is a’’ appear right before
Revised: (Label) per:origin (Explanation the words ‘‘is a’’ appear right before

Original: OBJ-
Revised: OBJ-

Original: NATIONALITY
Revised: NATIONALITY

Original: and
Revised: and

Original: the word
Revised: the word

Original: ‘‘citizen’’
Revised: ‘‘citizen’’

Original: is right after
Revised: is right after

Original: OBJ-NATIONALITY.
Revised: OBJ-NATIONALITY.

Original: SemEval 2010 Task
Revised: SemEval 2010 Task

Original: The SUBJ-O
Revised: The SUBJ-O

Original: is caused by the OBJ-O of UV radiation by the oxygen and ozone .
Revised: is caused by the OBJ-O of UV radiation by the oxygen and ozone .

Original: (Label) Cause-Effect(e2,e1)
Revised: (Label) Cause-Effect(e2,e1)

Original: (Explanation)
Revised: (Explanation)

Original: The phrase ‘‘is caused by the’’ occurs between SUBJ and OBJ and OBJ follows SUBJ.
Revised: The phrase ‘‘is caused by the’’ occurs between SUBJ and OBJ and OBJ follows SUBJ.

Original: SUBJ-O are parts of the OBJ-O OBJ-O disregarded by the compiler.
Revised: SUBJ-O are parts of the OBJ-O OBJ-O disregarded by the compiler.

Original: (Label)
Revised: (Label)

Original: Component-Whole(e1,e2)
Revised: Component-Whole(e1,e2)

Original: (Explanation)
Revised: (Explanation)

Original: The phrase ‘‘are parts of the’’ occurs between SUBJ and OBJ and OBJ follows SUBJ
Revised: The phrase ‘‘are parts of the’’ occurs between SUBJ and OBJ and OBJ follows SUBJ

Original: SemEval 2014 Task 4 - restaurant
Revised: SemEval 2014 Task 4 - restaurant

Original: I am relatively new to the area and tried Pick a bgel on 2nd and was disappointed with the service
Revised: I am relatively new to the area and tried Pick a bgel on 2nd and was disappointed with the service

Original: and I thought the food was overated and on the pricey side.
Revised: and I thought the food was overated and on the pricey side.

Original: (Term: food) (Label) negative (Explanation) the words ’overated’ is within 2 words after term
Revised: (Term: food) (Label) negative (Explanation) the words ’overated’ is within 2 words after term

Original: The decor is vibrant and eye-pleasing with several semi-private boths on the right side of the dining hall, which are great for a date.
Revised: The decor is vibrant and eye-pleasing with several semi-private boths on the right side of the dining hall, which are great for a date.

Original: (Term: decor) (Label) positive (Explanation) the term is followed by ’vibrant’ and ’eye-pleasing’
Revised: (Term: decor) (Label) positive (Explanation) the term is followed by ’vibrant’ and ’eye-pleasing’

Original: SemEval 2014 Task 4 - laptop
Revised: SemEval 2014 Task 4 - laptop

Original: It’s priced very reasonable and works very well right out of the box.
Revised: It’s priced very reasonable and works very well right out of the box.

Original: (Term: works) (Label) positive (Explanation) the word ‘‘resonable’’ occurs before term by no more
Revised: (Term: works) (Label) positive (Explanation) the word ‘‘resonable’’ occurs before term by no more

Original: than 2 words
Revised: than 2 words

Original: The DVD drive randomly pops open when it is in my backpack as well, which is annoying.
Revised: The DVD drive randomly pops open when it is in my backpack as well, which is annoying.

Original: (Term:
Revised: (Term:

Original: DVD drive) (Label) negative
Revised: DVD drive) (Label) negative

Original: (Explanation)
Revised: (Explanation)

Original: The word ’annoying’ occurs after term
Revised: The word ’annoying’ occurs after term

Original: A. I MPLEMENTATION D ETAILS
Revised: A. I MPLEMENTATION D ETAILS

Original: We use 300-dimensional word embeddings pre-trained by GloVe (Pennington et al., 2014).
Revised: We use 300-dimensional word embeddings pre-trained by GloVe (Pennington et al., 2014).

Original: The dropout rate for embeddings is 0.96 and the dropout rate for our sentence encoder is 0.5.
Revised: The dropout rate for embeddings is 0.96 and the dropout rate for our sentence encoder is 0.5.

Original: The hidden state size of the encoder is 300 and the hidden state size of the attention layer is 200.
Revised: The hidden state size of the encoder is 300 and the hidden state size of the attention layer is 200.

Original: We choose
Revised: We choose

Original: Adagrad as the optimizer and the learning rate for training classiﬁers is set to 0.5.
Revised: Adagrad as the optimizer and the learning rate for training classiﬁers is set to 0.5.

Original: For TACRED, in the pretraining stage, we set the learning rate to 0.1.
Revised: For TACRED, in the pretraining stage, we set the learning rate to 0.1.

Original: The total epochs for pretraining is 10.
Revised: The total epochs for pretraining is 10.

Original: The weight for L sim is set to 0.5.
Revised: The weight for L sim is set to 0.5.

Original: The batch size for pretraining is set to 100.
Revised: The batch size for pretraining is set to 100.

Original: For training the classiﬁer, the batch size for labeled data and unlabeled data is 50 and 100 respectively, the weight α for L u is set to 0.7, the weight β for L string is set to 0.2, the weight γ for L sim is set to 2.5.
Revised: For training the classiﬁer, the batch size for labeled data and unlabeled data is 50 and 100 respectively, the weight α for L u is set to 0.7, the weight β for L string is set to 0.2, the weight γ for L sim is set to 2.5.

Original: For SemEval 2010 Task 8, in the pretraining stage, we set the learning rate to 0.1.
Revised: For SemEval 2010 Task 8, in the pretraining stage, we set the learning rate to 0.1.

Original: The total epochs for pretraining is 10.
Revised: The total epochs for pretraining is 10.

Original: The weight for L sim is set to 0.5.
Revised: The weight for L sim is set to 0.5.

Original: The batch size for pretraining is set to 10.
Revised: The batch size for pretraining is set to 10.

Original: For training the classiﬁer, the batch size for labeled data and unlabeled data is 50 and 100 respectively, the weight α for L u is set to 0.5, the weight β for L string is set to 0.1, the weight γ for L sim is set to 2.
Revised: For training the classiﬁer, the batch size for labeled data and unlabeled data is 50 and 100 respectively, the weight α for L u is set to 0.5, the weight β for L string is set to 0.1, the weight γ for L sim is set to 2.

Original: For two datasets in SemEval 2014 Task 4, in the pretraining stage, we set the learning rate to 0.5.
Revised: For two datasets in SemEval 2014 Task 4, in the pretraining stage, we set the learning rate to 0.5.

Original: The total epochs for pretraining is 20.
Revised: The total epochs for pretraining is 20.

Original: The weight for L sim is set to 5.
Revised: The weight for L sim is set to 5.

Original: The batch size for pretraining is set to 20.
Revised: The batch size for pretraining is set to 20.

Original: For training the classiﬁer, the batch size for labeled data and unlabeled data is 10 and 50 respectively, the weight α for L u is set to 0.5, the weight β for L string is set to 0.1, the weight γ for L sim is set to 2.
Revised: For training the classiﬁer, the batch size for labeled data and unlabeled data is 10 and 50 respectively, the weight α for L u is set to 0.5, the weight β for L string is set to 0.1, the weight γ for L sim is set to 2.

Original: For ATAE-LSTM, we set hidden state of attention layer to be 300 dimension.
Revised: For ATAE-LSTM, we set hidden state of attention layer to be 300 dimension.

Original: A. F ULL R ESULTS
Revised: A. F ULL R ESULTS

Original: The full results for relation extraction and sentiment analysis are listed in Table 6 and Table 7 respectively.
Revised: The full results for relation extraction and sentiment analysis are listed in Table 6 and Table 7 respectively.

Original: [Table] [Table]
Revised: [Table] [Table]