Original: CLUES : A Benchmark for Learning Classifiers using Natural Language Explanations
Revised: CLUES : A Benchmark for Learning Classifiers using Natural Language Explanations

Original: Anonymous ACL submission

Original: Abstract
Revised: Abstract

Revised: Supervised learning has traditionally focused on inductive learning by observing labeled examples of a task.

Original: A hallmark of human intelligence is the abilityto learn new conceptspurely from language.
Revised: In contrast, humans have the ability to learn new concepts from language.

Original: In this work, we explore training classifiersfor structured data 1 purely from language. 
Revised: Here, we explore training classifiers for structured data 1 purely from language. We introduce

Original: Weintroduce CLUES , a benchmark for C lassifier 
Revised: CLUES , a benchmark for C lassifier L earning

Original: L earning U sing natural language E xplanation S ,consisting of a range of classification tasks overstructured data along with language supervisionin the form of explanations.
Revised: U sing natural language E xplanation S , consisting of a range of classification tasks over structured data along with language supervision in the form of explanations.

Original: CLUES consistsof 36 real-world and 144 synthetic classification tasks.
Revised: CLUES consists of 36 real-world and 144 synthetic classification tasks.

Original: It contains crowdsourced explanations describing real-world tasks from multipleteachers and programmatically generated explanations for the synthetic tasks.
Revised: It contains crowdsourced explanations describing real-world tasks from multiple teachers and programmatically generated explanations for the synthetic tasks.

Original: We develop
Revised: We develop

Original: ExEnt , an entailment-based model that learnsclassifiers using explanations.
Revised: ExEnt , an entailment-based model that learns classifiers using explanations.

Original: ExEnt generalizes up to 18% better (relative) on novel tasksthan a baseline that does not use explanations.
Revised: ExEnt generalizes up to 18% better (relative) on novel tasks than a baseline that does not use explanations.

Revised: Our code and datasets are available at: https: //clues-benchmark.github.io .

Original:  Introduction
Revised:  Introduction

Original: Humans have a remarkable ability to learn conceptsthrough language (Chopra et al., 2019; Tomasello,1999).
Revised: Humans have a remarkable ability to learn concepts through language (Chopra et al., 2019; Tomasello, 1999).

Original: For example, we can learn about poisonousmushrooms through an explanation like ‘a mushroom is poisonous if it has pungent odor’ .
Revised: For example, we can learn about poisonous mushrooms through an explanation like ‘a mushroom is poisonous if it has pungent odor’ .

Original: Suchan approach profoundly contrasts with the predominant paradigm of machine learning, wherealgorithms extract patterns by looking at scores oflabeled examples of poisonous and edible mushrooms.
Revised: Such an approach profoundly contrasts with the predominant paradigm of machine learning, where algorithms extract patterns by looking at scores of labeled examples of poisonous and edible mushrooms.

Original: However, it is unnatural to presume theavailability of labeled examples for the heavy tailof naturally occurring concepts in the world.
Revised: However, it is unnatural to presume the availability of labeled examples for the heavy tail of naturally occurring concepts in the world.

Original: This work studies how models trained to learnfrom natural language explanations can generalize to novel tasks without access to labeled examples.
Revised: This work studies how models trained to learn from natural language explanations can general- [Table] ize to novel tasks without access to labeled examples.

Original: While prior works in this area (Srivastavaet al., 2017, 2018; Hancock et al., 2018; Murtyet al., 2020, inter alia ) have explored explanationsas a source of supervision, they evaluate models on a small number of tasks (2-3 relation extraction tasks in Hancock et al.
Revised: While prior works in this area (Srivastava et al., 2017, 2018; Hancock et al., 2018; Murty et al., 2020, inter alia ) have explored explanations as a source of supervision, they evaluate models on a small number of tasks (2-3 relation extraction tasks in Hancock et al.

Original: ; Murty et al.
Revised: ; Murty et al.

Original: (2020), 7 email categorization tasks (Srivastava et al., 2017)).
Revised: (2020), 7 email categorization tasks (Srivastava et al., 2017)).

Original: Owing to the paucity of largescale benchmarks for learning from explanations over diverse tasks, we develop CLUES , a benchmark of classification tasks paired with natural language explanations.
Revised: Owing to the paucity of largescale benchmarks for learning from explanations over diverse tasks, we develop CLUES , a benchmark of classification tasks paired with natural language explanations.

Original: Our benchmark is divided into CLUES-Real and CLUES-Synthetic consisting of tasks from real-world (UCI, Kaggle, and
Revised: Our benchmark is divided into CLUES-Real and CLUES-Synthetic consisting of tasks from real-world (UCI, Kaggle, and

Original: Wikipedia) and synthetic domains respectively.
Revised: Wikipedia) and synthetic domains respectively.

Original: Explanations for CLUES-Real are crowdsourced to mimic the diversity and difficulty of human learning and pedagogy.
Revised: Explanations for CLUES-Real are crowdsourced to mimic the diversity and difficulty of human learning and pedagogy.

Original: For CLUES-Synthetic , we generate the explanations programmatically to test  models’ reasoning ability under a range of structural and linguistic modifications of explanations.
Revised: For CLUES-Synthetic , we generate the explanations programmatically to test [Figure] models’ reasoning ability under a range of structural and linguistic modifications of explanations.

Original: In addition to creating CLUES , we train models with a mix of explanations and labeled examples, in a multi-task setup, over a set of seen classification tasks to induce generalization to novel tasks, where we do not have any labeled examples.
Revised: In addition to creating CLUES , we train models with a mix of explanations and labeled examples, in a multi-task setup, over a set of seen classification tasks to induce generalization to novel tasks, where we do not have any labeled examples.

Original: However, we notice that simply concatenating explanations to the input does not help pre-trained models, like
Revised: However, we notice that simply concatenating explanations to the input does not help pre-trained models, like

Original: RoBERTa (Liu et al., 2019), generalize.
Revised: RoBERTa (Liu et al., 2019), generalize.

Original: Hence, we develop ExEnt , an entailment-based model for learning classifiers guided by explanations.
Revised: Hence, we develop ExEnt , an entailment-based model for learning classifiers guided by explanations.

Original: ExEnt shows a relative improvement of up to 18% over other baselines on novel tasks.
Revised: ExEnt shows a relative improvement of up to 18% over other baselines on novel tasks.

Original: Our contributions are:
Revised: Our contributions are:

Original: • We introduce CLUES , a benchmark for learning classifiers over structured data from language.
Revised: • We introduce CLUES , a benchmark for learning classifiers over structured data from language.

Original: • We develop ExEnt , an entailment-based model for learning classifiers guided by explanations.
Revised: • We develop ExEnt , an entailment-based model for learning classifiers guided by explanations.

Original:  Creating CLUES
Revised:  Creating CLUES

Original: In CLUES , we frame classification tasks over structured data represented in tabular format.
Revised: In CLUES , we frame classification tasks over structured data represented in tabular format.

Original: We consider two splits of our benchmark, CLUES-Real (real-world datasets) and CLUES-Synthetic (synthetic datasets).
Revised: We consider two splits of our benchmark, CLUES-Real (real-world datasets) and CLUES-Synthetic (synthetic datasets).

Original:  CLUES-Real
Revised: 2. CLUES-Real

Original: We first gather/create classification tasks from UCI,
Revised: We first gather/create classification tasks from UCI,

Original: Kaggle, and Wikipedia tables, then collect explanations for each classification task.
Revised: Kaggle, and Wikipedia tables, then collect explanations for each classification task.

Original: Collecting datasets: We choose 18 tabular classification tasks from UCI ML repository 2 and 7from Kaggle 3 .
Revised: Collecting datasets: We choose 18 tabular classification tasks from UCI ML repository 2 andfrom Kaggle 3 .

Original: Additionally, we mine suitable tables from Wikipedia that can be posed as classification tasks.
Revised: Additionally, we mine suitable tables from Wikipedia that can be posed as classification tasks.

Original: We formalize the mining processas a crowdsourcing task (details in Appendix C.2).
Revised: We formalize the mining process as a crowdsourcing task (details in Appendix C.2).

Original: We identified 11 classification tasks corresponding to 9 Wikipedia tables after mining around10K Wikipedia tables.
Revised: We identified 11 classification tasks corresponding to 9 Wikipedia tables after mining around 10K Wikipedia tables.

Original: The details of tasks in
Revised: The details of tasks in

Original: CLUES-Real are provided in Appendix C.
Revised: CLUES-Real are provided in Appendix C.

Original: Explanation CollectionPipeline:
Revised: Explanation Collection

Revised: Pipeline

Original:  1. C OLLECTING EXPLANATIONS :
Revised: : 1. C OLLECTING EXPLANATIONS :

Original: We use the
Revised: We use the

Original: Amazon Mechanical Turk (AMT) platform tocollect explanations for CLUES-Real .
Revised: Amazon Mechanical Turk (AMT) platform to collect explanations for CLUES-Real .

Original: In each
Revised: In each

Original: HIT, we provide turkers with a few labeled examples of a task and ask them to provide a setof explanations corresponding to the task.
Revised: HIT, we provide turkers with a few labeled examples of a task and ask them to provide a set of explanations corresponding to the task.

Original: Theturkers participating in this task have been vettedby a qualification task to test their understandingof ‘good’ and ‘bad’ explanations.
Revised: The turkers participating in this task have been vetted by a qualification task to test their understanding of ‘good’ and ‘bad’ explanations (see

Revised: Appendix H for templates).

Original: After providing explanations, the turkers predict classes for aset of unlabeled samples by using their explanations.
Revised: After providing explanations, the turkers predict classes for a set of unlabeled samples by using their explanations.

Original: The turkers in this stage are henceforthreferred as ‘teachers’ in our setup since theyprovide explanations to ‘teach’ models aboutdifferent classification tasks.
Revised: The turkers in this stage are henceforth referred as ‘teachers’ in our setup since they provide explanations to ‘teach’ models about different classification tasks.

Original: 2. E XPLANATION VERIFICATION:
Revised: 2. E XPLANATION VERIFICATION

Original: We validatethe utility of the explanations for a task fromeach teacher by evaluating if they are useful forother humans in learning the task.
Revised: We validate the utility of the explanations for a task from each teacher by evaluating if they are useful for other humans in learning the task.

Original: We performexplanation verification on AMT, where we aska new set of turker to label a set of unlabeledsamples from a task using the explanations provided by a teacher in the ‘explanation collection’phase.
Revised: We perform explanation verification on AMT as well, where we ask a new set of turkers to label a set of unlabeled samples from a task using the explanations provided by individual teachers in the ‘explanation collection’ phase.

Original: Additionally, we ask the turkers to givea Likert rating (1-4 scale) on the usefulness ofeach explanation.
Revised: Additionally, we ask the turkers to give a Likert rating (1-4 scale) on the usefulness of each explanation.

Original: Since the turkers in the verification stage perform the classification task usinglanguage explanations from a teacher, we referto them as ‘students’ for our setup henceforth.
Revised: Since the turkers in the verification stage perform the classification task using language explanations from a teacher, we refer to them as ‘students’ for our setup henceforth.

Original: 2. CLUES-Synthetic
Revised: [Figure] equen cy

Original: To delineate challenges in learning from explanations under controlled settings, we create
Revised: Likert Ratings of Explanations [Figure]

Original: [Table]
Revised: [Figure]

Original: CLUES-Synthetic , a set of programmatically cre-

Original: [Table]
Revised: [Figure]

Original: ated classification tasks with varying complexity

Original: [Table]
Revised: [Table]

Original: of explanations

Original: (in terms of structure and presence

Original: [Table]
Revised: [Figure]

Original: of quantifiers, conjunctions, etc.)
Revised: 2. CLUES-Synthetic

Original: and concept def- [Table] initions. We use the following types of rules that
Revised: To delineate challenges in learning from explanations under controlled settings, we create

Original: [Table] differ in structure and complexity ( c i denotes i th clause and l denotes a label): Simple: IF c 1 THEN l • Conjunctive: IF c 1 AND c 2 THEN l
Revised: CLUES-Synthetic , a set of programmatically created classification tasks with varying complexity of explanations (in terms of structure and presence of quantifiers, conjunctions, etc.)

Original: • Disjunctive: IF c 1 OR c 2 THEN l • Nested disjunction over conjunction: IF c 1 OR ( c 2 AND c 3 ) THEN l
Revised: and concept definitions.

Original: • Nested conjunction over disjunction: IF c

Original: AND ( c 2 OR c 3 ) THEN l • For each of the above, we include variants with negations (in clauses and/or labels)
Revised: We use the following types of rules that differ in structure and complexity ( c i denotes i th clause and l denotes a label):

Original: : for
Revised: Simple:

Original: example IF c 1 THEN NOT l
Revised: IF c 1 THEN l • Conjunctive: IF c 1 AND c 2 THEN l • Disjunctive: IF c 1 OR c 2 THEN l • Nested disjunction over conjunction: IF c 1 OR ( c 2 AND c 3 ) THEN l • Nested conjunction over disjunction: IF cAND ( c 2 OR c 3 ) THEN l • For each of the above, we include variants with negations (in clauses and/or labels): for example, IF c 1 THEN NOT l

Original: We also consider other linguistic variations of rules by inserting quantifiers (such as ‘always’, ‘likely’).
Revised: We also consider other linguistic variations of rules by inserting quantifiers (such as ‘always’, ‘likely’).

Original: The synthetic explanations are template-generated from the rules used in creating the task.
Revised: The synthetic explanations are template-generated from the rules used in creating the task.

Original: For brevity, we defer additional details on the use of quantifiers, label assignment using rules, and creation of synthetic explanations to Appendix B. Overall we have 48 different task types (based on the number of classes and rule variants) using which we synthetically create 144 classification tasks.
Revised: For brevity, we defer additional details on the use of quantifiers, label assignment using rules, and creation of synthetic explanations to Appendix B. Overall we have 48 different task types (based on the number of classes and rule variants) using which we synthetically create 144 classification tasks.

Original:  Dataset analysis
Revised:  Dataset analysis

Original: In this section, we analyze the tasks and the collected explanations in CLUES .
Revised: In this section, we analyze the tasks and the collected explanations in CLUES .

Original: Task Statistics :
Revised: Task Statistics :

Original: Table 1 shows the statistics of tasks in CLUES .
Revised: Table 1 shows the statistics of tasks in CLUES .

Original: Task details are provided in Appendices B and C. An aggregate of 133 teachers provide 318 explanations for tasks in CLUES-Real .
Revised: Task details are provided in Appendices B and C. An aggregate of 133 teachers provide 318 explanations for tasks in CLUES-Real .

Original: All collected explanations were manually filtered and irrelevant explanations were removed.
Revised: All collected explanations were manually filtered and irrelevant explanations were removed.

Original: Further, each explanation set corresponding to a teacher in
Revised: Further, each explanation set corresponding to a teacher in

Original: CLUES-Real was verified by 3 students.
Revised: CLUES-Real was verified by 3 students.

Original: Lexical analysis of explanations : Using the spacy tokenizer, we find the vocabulary size of

Original: [Figure]

Original: he r -

Original: [Figure]

Original: v g .

Original: T ea c he r

Original: A cc u r a cy

Original: [Figure]

Original: Teacher

Original: [Figure]

Original: Avg.

Original: Teacher) vs (Student

Original: [Figure]

Original: Avg.

Original: Student) best

Original: [Figure] explanations in CLUES as 1026 tokens resulting[Figure] in 15.53 tokens on average per explanation.
Revised: Lexical analysis of explanations : Using the spacy tokenizer, we find the vocabulary size of explanations in CLUES as 1026 tokens resulting in 15.53 tokens on average per explanation.

Original: The

Original: [Figure]

Original:  median reading complexity of the explanations is [Figure] [Figure] 73 (8th/9th-grade reading level) 
Revised: The median reading complexity of the explanations is 65.73 (8th/9th-grade reading level) 5 .

Original: [Figure]

Original: Usefulness of the explanations : During explanation verification, turkers provide a rating (on a
Revised: Usefulness of the explanations : During explanation verification, turkers provide a rating (on a

Original: Likert scale from 1 to 4) on the utility of the explanations for classification (1 - ‘not helpful’ ; 4 -‘denotes mostly helpful in prediction’).
Revised: Likert scale from 1 to 4) on the utility of the explanations for classification (1 - ‘not helpful’ ; 4 - ‘denotes mostly helpful in prediction’).

Original: The average rating for the explanations in CLUES-Real is2.78, denoting most explanations were useful.
Revised: The average rating for the explanations in CLUES-Real is 2.78, denoting most explanations were useful.

Original: Characteristics of teachers and students : Figure 1 shows the normalized teacher performancevs normalized student performance for teacherstudent pairs in CLUES-Real .
Revised: Characteristics of teachers and students : Figure 3(b) shows the normalized teacher performance vs normalized student performance for teacherstudent pairs in CLUES-Real .

Original: Normalized performance of an individual teacher (or, student) on atask is defined as the difference between the performances of the teacher (or, student) and an averageteacher (or, student) for the same task.
Revised: Normalized performance of an individual teacher (or, student) on a task is defined as the difference between the performances of the teacher (or, student) and an average teacher (or, student) for the same task.

Original: The positivecorrelation ( ρ = 0.17) suggests that students tend  to perform well if taught by well-performing teachers.
Revised: The positive correlation ( ρ = 0.17) suggests that students tend [Figure] teacherstudentrandom to perform well if taught by well-performing teachers.

Original: Figure 2 shows average teacher and avergage student performance for each task in CLUES-Real .
Revised: Figure 4 shows average teacher and avergage student performance for each task in CLUES-Real .

Original: We find that an average teacher performs better than the average student on most tasks, barring a few tasks such as ‘tic-tac-toe’, which could be solved using commonsense without relying on explanations.
Revised: We find that an average teacher performs better than the average student on most tasks, barring a few tasks such as ‘tic-tac-toe’, which could be solved using commonsense without relying on explanations.

Original: Please refer to Appendix D for additional data analysis and insights on CLUES .
Revised: Please refer to Appendix D for additional data analysis and insights on CLUES .

Original:  Experiment Setup and Models
Revised:  Experiment Setup and Models

Original: In this section, we describe our training and evaluation setup, our models, and experimental findings.
Revised: In this section, we describe our training and evaluation setup, our models, and experimental findings.

Original: 4. Training and Evaluation Setup
Revised: 4. Training and Evaluation Setup

Original: Our goal is to learn a model that, at inference, can perform classification over an input x to obtain the class label y , given the set of explanations E for the classification task.
Revised: Our goal is to learn a model that, at inference, can perform classification over an input x to obtain the class label y , given the set of explanations E for the classification task.

Original: We train our model using multitask training over a set of tasks T seen and evaluate generalization to a new task, t ∈ T novel .
Revised: We train our model using multitask training over a set of tasks T seen and evaluate generalization to a new task, t ∈ T novel .

Original: The task split we use for our experiments can be found in
Revised: The task split we use for our experiments can be found in

Original: Appendix F.1.
Revised: Appendix F.1.

Original: We select our best model for zeroshot evaluation based on the validation scores on the seen tasks.
Revised: We select our best model for zeroshot evaluation based on the validation scores on the seen tasks.

Original: Following linearization techniques for structured data Yin et al.
Revised: Following linearization techniques for structured data Yin et al.

Original: (2020), we adapt a similar encoding process wherein we encode each structured data example, x , as a text sequence, by linearizing it as a sequence of attribute-name and attribute-value pairs, separated by [SEP] tokens.
Revised: (2020), we adapt a similar encoding process wherein we encode each structured data example, x , as a text sequence, by linearizing it as a sequence of attribute-name and attribute-value pairs separated by [SEP] tokens.

Original: We will refer to the linearized format of structured inputs by ‘Features-as-Text’ or ‘FaT’.
Revised: We will refer to the linearized format of structured inputs by ‘Features-as-Text’ or ‘FaT’.

Original: 4. Baseline models
Revised: 4. Baseline models

Original: We consider the following baselines:
Revised: We consider the following baselines:

Original: • RoBERTa w/o Exp.
Revised: • RoBERTa w/o Exp.

Original: : This is a RoBERTa- based (Liu et al., 2019) baseline that does not utilize the explanations to train a classifier.The
Revised: : This is a RoBERTabased (Liu et al., 2019) baseline that does not utilize the explanations to train a classifier.

Original:  pre-trained RoBERTa model takes the linearized structured data (FaT) as input and outputs a rep- resentation for this context (using the [CLS] token).
Revised: The pre-trained RoBERTa model takes the linearized structured data (FaT) as input and outputs a representation for this context (using the [CLS] token).

Original: Next, we run another forward pass using RoBERTa to obtain a representation of the labels based on their text.
Revised: Next, we run another forward pass using RoBERTa to obtain a representation of the labels based on their text.

Original: We compute the probability distribution over labels by doing a dot-product of the representations of the input and the labels.
Revised: We compute the probability distribution over labels by doing a dot-product of the representations of the input and the labels.

Original: We train this model using cross-entropy loss.
Revised: We train this model using cross-entropy loss.

Original: • RoBERTa w/ Exp.
Revised: • RoBERTa w/ Exp.

Original: : This follows the same train- ing setup as RoBERTa w/o Exp.
Revised: : This follows the same training setup as RoBERTa w/o Exp.

Original: However, in addition to FaT, this model also takes a concate- nation of explanations as the input to RoBERTa.MLM
Revised: However, in addition to FaT, this model also takes a concatenation of explanations as the input to RoBERTa.

Original: Exp.

Original: FaT [SEP

Original: [Figure]

Original: [SEP] [CLS

Original: [Figure]

Original: Exp

Original: [Figure]

Original: FaT [SEP

Original: [Figure]

Original: [SEP] [CLS

Original: [Figure]

Original: Entailment

Original: [Figure]

Original: Agg.

Original: [Figure]

Original: [Figure]

Original: 4. ExEnt
Revised: 4. ExEnt

Original: We empirically notice that simply concatenatingexplanations to the input does not help pre-trainedmodels, like RoBERTa generalize.
Revised: We empirically notice that simply concatenating explanations to the input does not help pre-trained models, like RoBERTa generalize.

Original: In order to better model the influence of an explanation towardsdeciding a class label, we draw analogies with theentailment of an explanation ( hypothesis ) towardsthe structured input ( premise ).
Revised: In order to better model the influence of an explanation towards deciding a class label, we draw analogies with the entailment of an explanation ( hypothesis ) towards the structured input ( premise ).

Original: Figure 3 shows theoverview of our explanation-guided classificationmodel, ExEnt .
Revised: Figure 5 shows the overview of our explanation-guided classification model, ExEnt .

Original: Given a structured input and explanation of a task, let l exp denote the label mentionedin the explanation, and L denote the set of labelsof the task.
Revised: Given a structured input and explanation of a task, let l exp denote the label mentioned in the explanation, and L denote the set of labels of the task.

Original: The entailment model provides entailment ( p e ), contradiction ( p c ) and neutral ( p n ) logitswhich are then assigned to the class labels as:
Revised: The entailment model provides entailment ( p e ), contradiction ( p c ) and neutral ( p n ) logits which are then assigned to the class labels as:

Original: • If explanation mentions to assign a label :
Revised: • If explanation mentions to assign a label :

Original: As- sign p e to l exp , p c is divided equally among labels in L \ { l exp } , and p n is divided equally among labels in L .
Revised: Assign p e to l exp , p c is divided equally among labels in L \ { l exp } , and p n is divided equally among labels in L .

Original: • If explanation mentions to not assign a label : This occurs if a negation is associated with l exp .
Revised: • If explanation mentions to not assign a label : This occurs if a negation is associated with l exp .

Original: Assign p c to l exp , p e is divided equally amonglabels in L \ { l exp } , and p n is divided equallyamong labels in L .
Revised: Assign p c to l exp , p e is divided equally among labels in L \ { l exp } , and p n is divided equally among labels in L .

Original: We obtain logit scores over labels of the task corresponding to each explanation as described above.
Revised: We obtain logit scores over labels of the task corresponding to each explanation as described above.

Original: We compute the final label logits by aggregating(using mean) over the label logits corresponding toeach explanation of the task.
Revised: We compute the final label logits by aggregating (using mean) over the label logits corresponding to each explanation of the task.

Original: The final label logitsare converted to a probability distribution over labels, and we train ExEnt using cross-entropy loss.
Revised: The final label logits are converted to a probability distribution over labels, and we train ExEnt using cross-entropy loss.

Original: Additional training details and hyperparameters are provided in Appendix F.
Revised: MLM Exp. FaT [SEP] [SEP] [CLS]

Revised: Exp.

Original: 4. Zero-Shot Generalization Performance
Revised: FaT [SEP] [SEP] [CLS] Exp. FaT [SEP] [SEP] [CLS] Entailment Logits

Revised: Agg.

Original:  Figure 4 shows the generalization performance of all models on CLUES .
Revised: [Figure] [Figure] 4. Zero-Shot Generalization Performance Figure 6 shows the generalization performance of all models on CLUES .

Original: ExEnt outperforms the baselines suggesting that performing entailment as an intermediate step helps aggregate information from multiple explanations better.
Revised: ExEnt outperforms the baselines suggesting that performing entailment as an intermediate step helps aggregate information from multiple explanations better.

Original: On CLUES-Real ,
Revised: On CLUES-Real ,

Original: ExEnt gets an 18% relative improvement over the baselines while having an 11% relative improvement on CLUES-Synthetic .
Revised: ExEnt gets an 18% relative improvement over the baselines while having an 11% relative improvement on CLUES-Synthetic .

Original: Additionally, we find that pre-training on synthetic tasks boosts the generalization performance of  ExEnt  on CLUES-Real by 30% (relative) over the baselines.
Revised: To evaluate the utility of our synthetic tasks in enabling transfer learning to real-world tasks, we fine-tune a ExEnt model pre-trained on synthetic tasks. We experiment with three pre-training task sets - CLUES-Synthetic , CLUES-Synthetic (3x) and CLUES-Synthetic (5x) consisting of 144, 432, and 720 tasks.

Original:  We hypothesize that ExEnt is able to learn better initialization on account of being trained on a large suite of tasks with aligned goals enabling it to generalize on the smaller collection of real-world tasks easily.
Revised: These larger synthetic task sets are created by sampling tasks from each of the 48 different synthetic tasks types similar to how CLUES-Synthetic was created (see §2.2 for reference). We find that pretraining on synthetic tasks boosts the performance of ExEnt on the novel tasks of CLUES-Real by up to 39% (relative) over the RoBERTa w/o Exp. model.

Original: Additional experiments on synthetic tasks also reveal that ExEnt struggles with handling negations, conjunctions and disjunctions when learning from explanations (details in Appendix G). 
Revised: Additional experiments on synthetic tasks also reveal that ExEnt struggles with handling negations, conjunctions and disjunctions when learning from explanations (details in Appendix G). Human Performance

Revised: To situate the performance of the automated models, we performed human evaluation for tasks in test split of

Revised: CLUES-Real using AMT.

Revised: For this, we sampled at most 50 examples 6 from the test split of tasks in

Original: Conclusion
Revised: CLUES-Real and each example was ‘labeled’ byturkers using the explanations of the ‘best teacher’ (the teacher whose students got the best performance during ‘explanation verification’ stage; see §2 for reference). The average human accuracy for this was about 70%. However, the performance numbers of humans and models are not directly comparable as the model looks at all the explanations for the task, whereas the humans observe a small number of explanations. Humans also see multiple examples of the task during the evaluation, which they can use to fine-tune their understanding of a concept.

Revised: The automated models don’t have a mechanism to leverage such data.

Revised:  Conclusion

Original: We introduce CLUES , a benchmark with diverse classification tasks over structured data using natural language explanations to test the ability of models to learn novel classification tasks purely from language.
Revised: We introduce CLUES , a benchmark with diverse classification tasks over structured data using natural language explanations to test the ability of models to learn novel classification tasks purely from language.

Original: Additionally, we introduce ExEnt , an entailment-based model to learn classifiers guided by explanations.
Revised: Additionally, we introduce ExEnt , an entailment-based model to learn classifiers guided by explanations.

Original: Our results indicate that explicitly modeling the role of each explanation through entailment can enable learning classifiers for new tasks from explanations.
Revised: Our results indicate that explicitly modeling the role of each explanation through entailment can enable learning classifiers for new tasks from explanations.

Original: Future work can explore open challenges such as modeling quantifiers and negations present in an explanation.
Revised: Future work can explore open challenges such as modeling quantifiers and negations present in an explanation.

Original: CLUES is agnostic in the domain of tasks allowing the research community to contribute more tasks in the future.
Revised: CLUES is agnostic in the domain of tasks allowing the research community to contribute more tasks in the future.

Original: A Code and Models
Revised: A Code and Models

Original: Code and models for CLUES are submitted as supplementary material and will be publicly releasedupon first publication.
Revised: Code and models for CLUES are submitted as supplementary material and will be publicly released upon first publication.

Original: B Additional details on creatingCLUES-Synthetic
Revised: B Additional details on creating CLUES-Synthetic

Original: In this section we discuss in detail about the varioustable schemas followed by the details of quantifiersand label assignment for creating synthetic tasks.
Revised: In this section we discuss in detail about the various table schemas followed by the details of quantifiers and label assignment for creating synthetic tasks.

Original: B. Tables schemas
Revised: B. Tables schemas

Original: We define five different table schemas, each corresponding to a different domain.
Revised: We define five different table schemas, each corresponding to a different domain For

Original: For all the attributes in a schema we define a fixed domain fromwhich values forthat attribute can be sampled.
Revised: all the attributes in a schema we define a fixed domain from which values for

Revised: that attribute can

Original:  • Species of bird : The classification task here is to classify a bird into a particular species based on various attributes (column names in table).
Revised: be sampled • Species of bird : The classification task here is to classify a bird into a particular species based on various attributes (column names in table).

Original: We define several artificial species of birds us- ing commonly used nonce words in psycholog- ical studies (Chopra et al., 2019) such as “dax",“wug", etc.• Species of animal :
Revised: We define several artificial species of birds using commonly used nonce words in psychological studies (Chopra et al., 2019) such as “dax", “wug", etc. • Species of animal :

Original: The classification task here is to classify an animal into a particular species based on various attributes (column names intable).
Revised: The classification task here is to classify an animal into a particular species based on various attributes (column names in table).

Original: Artificial species of animals are againdefined using commonly used nonce words in psychological studies such as “dax", “wug", etc.
Revised: Artificial species of animals are again defined using commonly used nonce words in psychological studies such as “dax", “wug", etc.

Original: • Rainfall prediction : This is a binary classifica- tion task where the objective is to predict whetherit will rain tomorrow based on attributes such as“location", “minimum temperature", “humidity",“atmospheric pressure" etc.• Rank in league : This is a multi-label classification task where given attributes such “win percentage", “power rating", “field goal rating" ofa basketball club, the objective is to predict itsposition in the league out of 1, 2, 3, 4, "Not qualified".• Bond relevance : This is a multi-label classification task where given attributes such “user age",“user knowledge", “user
Revised: • Rainfall prediction : This is a binary classification task where the objective is to predict whether it will rain tomorrow based on attributes such as “location", “minimum temperature", “humidity", “atmospheric pressure" etc. • Rank in league : This is a multi-label classification task where given attributes such “win percentage", “power rating", “field goal rating" of a basketball club, the objective is to predict its position in the league out of 1, 2, 3, 4, "Not qualified".

Original:  income", the objective isto predict the relevance of a bond out of 5 classes(1 to 5).
Revised: • Bond relevance : This is a multi-label classification task where given attributes such “user age", “user knowledge", “user income", the objective is to predict the relevance of a bond out of 5 classes (1 to 5).

Original: In each of the above schemas, the attributes can beeither of types categorical or numeral.
Revised: In each of the above schemas, the attributes can be either of types categorical or numeral.

Original: For each ofthe above schemas we also define range of admissible values for each attribute.
Revised: For each of the above schemas we also define range of admissible values for each attribute.

Original: Detailed descriptionof schemas are provided in Tables 5, 6, 7 8, 9.
Revised: Detailed description of schemas are provided in Tables 5, 6, 7 8, 9.

Original: B. List of quantifiers
Revised: B. List of quantifiers

Original: The full list of quantifiers along with their associ- [Table] ated probability values are shown in Table 2.[Table]
Revised: The full list of quantifiers along with their associated probability values are shown in Table 2.

Original: [Table]
Revised: [Table]

Original: B. Creating synthetic explanations
Revised: B. Creating synthetic explanations

Original: [Table]

Original: We use a template-based approach to convert the[Table] set to rules into language explanations.We convert
Revised: We use a template-based approach to convert the set to rules into language explanations.

Original: [Table]

Original:  every operator in the clauses into their correspond- [Table] ing language format as:[Table]
Revised: We convert every operator in the clauses into their corresponding language format as:

Original: • ==
Revised: • ==

Original: → ‘equal to’ • > → ‘greater than’ • > =
Revised: → ‘equal to’ • > → ‘greater than’ • > =

Original: → ‘greater than or equal to’[Table] • < → ‘lesser than’ 
Revised: → ‘greater than or equal to’ • < → ‘lesser than’ • < = → ‘lesser than or equal to’ • !

Original: • <

Original: =
Revised: =

Original: → ‘lesser than or equal to’ 
Revised: → "not equal to’ • ! >

Original: [Table] = → "not equal to’ • ! > → ‘not greater than’ 
Revised: → ‘not greater than’ • !

Original: < → ‘not lesser than’
Revised: < → ‘not lesser than’

Original: For example if we have a rule ‘ IF number of hands =
Revised: For example if we have a rule ‘ IF number of hands =

Original: = 2 THEN foo ’
Revised: = 2 THEN foo ’

Original: , we convert it into a language explanation as ‘If number of hands equal to 2, then foo’.
Revised: , we convert it into a language explanation as ‘If number of hands equal to 2, then foo’.

Original: In the presence of quantifiers, we add ‘it is [INSERT QUANTIFIER] ’ before the label.
Revised: In the presence of quantifiers, we add ‘it is [INSERT QUANTIFIER] ’ before the label.

Original: For example if the rule was associated with a quantifier ‘usually’, the language explanation would be ‘If number of hands equal to 2, then it is usually foo’.
Revised: For example if the rule was associated with a quantifier ‘usually’, the language explanation would be ‘If number of hands equal to 2, then it is usually foo’.

Original: B. Label Assignment using Rules
Revised: B. Label Assignment using Rules

Original: In Algorithm 1, we detail the procedure for obtaining label assignments for our synthetic tasks.
Revised: In Algorithm 1, we detail the procedure for obtaining label assignments for our synthetic tasks.

Original: Given that our rules are in an “ IF ... THEN .." format, we split each rule into an antecedent and a consequent based on the position of THEN .
Revised: Given that our rules are in an “ IF ... THEN .." format, we split each rule into an antecedent and a consequent based on the position of THEN .

Original: Note that our voting-based approach to choose the final label for an example helps to tackle (1) negation on a label for multiclass tasks and (2) choose the most suited label in case antecedents from multiple rules are satisfied by an example.
Revised: Note that our voting-based approach to choose the final label for an example helps to tackle (1) negation on a label for multiclass tasks and (2) choose the most suited label in case antecedents from multiple rules are satisfied by an example.

Original: odor spore-print-color gill-color ring-type stalk-surface-abovering poisonous/edible none green gray pendant smooth poisonous none black black evanescent smooth edible pungent black white pendant smooth poisonous Explanations: ● Mushrooms with pungent or foul odors are poisonous . ● Mostly edible if the stalk-surface-above-ring is smooth.
Revised: Algorithm

Original: Explanation: ● If arms equal to yes and hair not equal to no, then fem . ● It venomous not equal to no and arms not equal to no, then not gazzer head hair arms legs venomous animal species yes yes yes yes fem
Revised: 1: Given: Task T with rule set R and label set L 2: Votes ← Zeros( | L | ) 3: for rule r ∈ R do 4: r a : Antecedent of r 5: r c : Consequent of r 6: l r ← Label mentioned in r c 7: t ← Truth Value of r a 8: if any quantifier in r then 9: p quant : Prob.

Original: no
Revised: of quantifier from Table 2 10: Alter l r to any label in L \ l r with probability 11: 1 − p quant 12: end if 13: if t =

Original: yes yes yes tupa
Revised: True then 14:

Original: no no
Revised: Votes[ l r ] += 1 15: else 16: for label l ∈ L \ l r do 17:

Original: yes no gazzer (a) (b)
Revised: Votes[ l ] += 1 18: end for 19: end if 20: l assigned ← argmax(Votes) 21: end for

Original: B. Different synthetic task types
Revised: B. Different synthetic task types

Original: We create our synthetic tasks by varying along thefollowing axes:
Revised: We create our synthetic tasks by varying along the following axes:

Original: • Number of labels: L = {‘binary’, ‘multiclass’ } • Structure of explanation: C = {‘simple’, ‘con- junction/disjunction’, ‘nested’ } • Presence of quantifier: Q = {‘not present’, 
Revised: • Number of labels: L = {‘binary’, ‘multiclass’ } • Structure of explanation: C = {‘simple’, ‘conjunction/disjunction’, ‘nested’ } • Presence of quantifier: Q = {‘not present’, ‘present’} • Negation: N = {‘no negation’, ‘negation only in clause’, ‘negation only on label’, ‘negation in clause or on label’}

Original: ‘present’}

Original: • Negation: N = {‘no negation’, ‘negation only

Original: in clause’, ‘negation only on label’, ‘negation inclause or on label’}

Original: The set of task types is defined as L × C × Q × N ,enumerating to 48 different types.
Revised: The set of task types is defined as L × C × Q × N , enumerating to 48 different types.

Original: C Real-World Tasks from UCI, Kaggleand Wikipedia
Revised: C Real-World Tasks from UCI, Kaggle and Wikipedia

Original: For our benchmark, we made use of 18 datasetsin UCI, 7 datasets in Kaggle, and 9 tables in
Revised: For our benchmark, we made use of 18 datasets in UCI, 7 datasets in Kaggle, and 9 tables in

Original: Wikipedia.
Revised: Wikipedia.

Original: In Table 4, we list the keywords thatwe use to refer to these tasks along with the URLsto the datasets/tables.
Revised: In Table 4, we list the keywords that we use to refer to these tasks along with the URLs to the datasets/tables.

Original: C. Selecting tasks from UCI and Kaggle
Revised: C. Selecting tasks from UCI and Kaggle

Original: We manually filter the available datasets to avoid ones with (a) many missing attributes and (b) complex attribute names that require extensive domain knowledge making them unsuitable for learning purely from language.
Revised: We manually filter the available datasets to avoid ones with (a) many missing attributes and (b) complex attribute names that require extensive domain knowledge making them unsuitable for learning purely from language.

Original: During pilot studies for collection of explanations for CLUES-Real , we identified that annotators found it difficult to provide explanations for classifications tasks with more than 5 to 6 columns.
Revised: During pilot studies for collection of explanations for CLUES-Real , we identified that annotators found it difficult to provide explanations for classifications tasks with more than 5 to 6 columns.

Original: Appropriately, we reduced the number of columns in most datasets of CLUES-Real to 5 by choosing the top features that had maximum mutual information with the labels in the training dataset.
Revised: Appropriately, we reduced the number of columns in most datasets of CLUES-Real to 5 by choosing the top features that had maximum mutual information with the labels in the training dataset.

Original: The mutual information between the features and the label was computed using the scikit-learn package with a random state of 624.
Revised: The mutual information between the features and the label was computed using the scikit-learn package with a random state of 624.

Original: C. Mining tables from Wikipedia
Revised: C. Mining tables from Wikipedia

Original: Wikipedia is a rich, free source of information readily accessible on the web with a lot of this information stored in a structured format in form of tables.
Revised: Wikipedia is a rich, free source of information readily accessible on the web with a lot of this information stored in a structured format in form of tables

Original: We explore creating additional classification tasks based on tables from Wikipedia, where each row in a table is assigned a category label.
Revised: We explore creating additional classification tasks based on tables from Wikipedia, where each row in a table is assigned a category label.

Original: However, only a small fraction of the tables might be suitable to frame a classification task for our benchmark.
Revised: However, only a small fraction of the tables might be suitable to frame a classification task for our benchmark.

Original: Thus, we need to identify suitable tables by mining a large collection of tables from Wikipedia (we use Wikipedia dump available on April 2021).
Revised: Thus, we need to identify suitable tables by mining a large collection of tables from Wikipedia (we use Wikipedia dump available on April 2021).

Original: We formalize this mining-and-pruning process as a crowdsourcing task (on Amazon Mechanical Turk), where we present each turker with a batch of tables and ask them to pick out suitable tables from that batch.
Revised: We formalize this mining-and-pruning process as a crowdsourcing task (on Amazon Mechanical Turk), where we present each turker with a batch oftables and ask them to pick out suitable tables from that batch.

Original: For a table considered suitable by a turker, we further ask the turker to mention which column of the table should be considered as providing the classification labels.
Revised: For a table considered suitable by a turker, we further ask the turker to mention which column of the table should be considered as providing the classification labels.

Original: D Additional Analysis on 
Revised: D Additional Analysis on Teacher-Student Performance

Original: Teacher-Student Performance

Original: Figure 8(a) shows the histogram of the Likert ratings (scale 1-4) provided by the students in theexplanation verification stage.
Revised: Figure 3(a) shows the histogram of the Likert ratings (scale 1-4) provided by the students in the explanation verification stage.

Original: The average ratingfor the explanations in CLUES-Real is 2.78, denoting most explanations were useful, even if they didnot directly help predict labels in some cases.
Revised: The average rating for the explanations in CLUES-Real is 2.78, denoting most explanations were useful, even if they did not directly help predict labels in some cases.

Original: Figure 8(b) shows average teacher and average studentperformance on tasks in CLUES-Real .
Revised: Figure 3(c) shows average teacher and average student performance on tasks in CLUES-Real .

Original: Positive correlation ( ρ = 0.48) in this figure indicates that taskdifficulty (captured by classification accuracy) iswell-correlated for a teacher and student on average.
Revised: Positive correlation ( ρ = 0.48) in this figure indicates that task difficulty (captured by classification accuracy) is well-correlated for a teacher and student on average.

Original: For the crowdsourced datasets, we show the number of explanations collected per task in Figure 9(a).
Revised: For the crowdsourced datasets, we show the number of explanations collected per task in Figure 9(a).

Original: The number of explanations is largely around anaverage value of 11 explanations per task.
Revised: The number of explanations is largely around an average value of 11 explanations per task.

Original: Figure 9(b) shows the relation between explanation quality (quantified by likert scores) and rank ofthe explanation.
Revised: Figure 9(b) shows the relation between explanation quality (quantified by likert scores) and rank of the explanation.

Original: Rank denotes the order in which ateacher provided that explanation during our crowdsourced explanation collection phase.
Revised: Rank denotes the order in which a teacher provided that explanation during our crowdsourced explanation collection phase.

Original: We find apositive correlation between quality and rank ofexplanation showing that teachers generally submitmost useful explanations (as perceived by them) toteach a task.
Revised: We find a positive correlation between quality and rank of explanation showing that teachers generally submit most useful explanations (as perceived by them) to teach a task.

Original: Finally, we do not observe any correlation between explanation length and ratings asindicated by Figure 9(c).
Revised: Finally, we do not observe any correlation between explanation length and ratings as indicated by Figure 9(c).

Original: We also illustrate the differences betweenteacher and student on our tasks in §3.
Revised: We also illustrate the differences between teacher and student on our tasks in §3.

Original: Here wepresent two additional plots showing the performance of (1) best teacher vs their students for eachtask (Figure 6) and (2) worst teacher vs their students for each task (Figure 7).
Revised: Here we present two additional plots showing the performance of (1) best teacher vs their students for each task (Figure 7) and (2) worst teacher vs their students for each task (Figure 8).

Original: We find that eventhough the best teachers often attain near-perfectaccuracies for the tasks, their students perform significantly worse than them in many tasks.
Revised: We find that even though the best teachers often attain near-perfect accuracies for the tasks, their students perform significantly worse than them in many tasks.

Original: Theexplanations from the worst teachers did not helpstudents in getting significantly better than randomperformance for majority of the tasks, even thoughthe student did outperform the worst teacher. 
Revised: The explanations from the worst teachers did not help students in getting significantly better than random performance for majority of the tasks, even though the student did outperform the worst teacher. [Figure]

Original: E Reward Structure for Crowd-sourcing 
Revised: E Reward Structure for Crowd-sourcing Tasks

Original: Tasks

Original: Our work involves multiple stages of crowdsourcing to collect high-quality explanations for the classification tasks.
Revised: Our work involves multiple stages of crowdsourcing to collect high-quality explanations for the classification tasks.

Original: We pick turkers in the US for explanation collection and verification tasks (US,UK,NZ,and GB for Wikipedia mining Task) with a 98%
Revised: We pick turkers in the US for explanation collection and verification tasks (US,UK,NZ, and GB for Wikipedia mining Task) with a 98%

Original: HIT approval rate and a minimum of 1000 HITs[Figure] [Figure] approved.
Revised: HIT approval rate and a minimum of 1000 HITs approved.

Original: In Table 3, we summarize the payment structure provided to the turkers on the AMT platform for each of the stages (described in detail in § 2) – (1) Wikipedia mining on tables scraped from 
Revised: In Table 3, we summarize the payment structure provided to the turkers on the AMT platform for each of the stages (described in detail in §2) – (1) Wikipedia mining on tables scraped from [Figure]

Original: Wikipedia, (2) Explanation collection for tables obtained from UCI, Kaggle and Wikipedia, and (3) Explanation validation for collected explana- tions.
Revised: Wikipedia, (2) Explanation collection for tables obtained from UCI, Kaggle and Wikipedia, and (3) Explanation validation for collected explanations.

Original: For all the three crowdsourcing tasks, the turkers were compensated fairly and the payment per task is equivalent to an hourly compensation that is greater than minimum wage (based on the 
Revised: For all the three crowdsourcing tasks, the turkers were compensated fairly and the payment per task is equivalent to an hourly compensation that is greater than minimum wage (based on the median time taken by turkers).

Original: median

Original: time taken by turkers

Original: [Figure] [Figure]
Revised: [Table]

Original: equen cy

Original: Likert Ratings of Explanations

Original: [Figure]
Revised: [Figure]

Original: age

Original: [Figure] he [Figure] an
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: sk

Original: [Figure]
Revised: [Table]

Original: Average Teacher

Original: [Figure] Student Performance Correlation [Figure] [Figure]
Revised: ExplanationCollection

Original: [Table]
Revised: [Table]

Original: [Figure] [Table] [Figure] [Table]
Revised: ExplanationValidation $1.

Original: F Training details
Revised: F Training details

Original: In this section we proved details about implementation of various models, hyperparameter details,and details about hardware and software used alongwith an estimate of time taken to train the models.
Revised: In this section we proved details about implementation of various models, hyperparameter details, and details about hardware and software used along with an estimate of time taken to train the models.

Original: Code and dataset for our paper will be made publicupon first publication.
Revised: Code and dataset for our paper will be made public upon first publication.

Original: F. Details of seen and novel tasks for 
Revised: F. Details of seen and novel tasks for CLUES-Real and CLUES-Synthetic

Original: CLUES-Real and CLUES-Synthetic

Original: For CLUES-Real , we chose the tasks from
Revised: For CLUES-Real , we chose the tasks from

Original: Wikipedia that have very examples to be part of novel task set.
Revised: Wikipedia that have very examples to be part of novel task set.

Original: Among the tasks from Kaggle and
Revised: Among the tasks from Kaggle and

Revised: [Figure]

Revised: e

Original:  UCI, we kept tasks with higher number of samples as part of seen tasks (training tasks).
Revised: [Figure] c [Figure] UCI, we kept tasks with higher number of samples as part of seen tasks (training tasks).

Original: Seen tasks (20) for CLUES-Real are:
Revised: Seen tasks (20) for CLUES-Real are:

Original: • website-phishing • internet-firewall • mushroom • dry-bean 
Revised: • website-phishing • internet-firewall • mushroom • dry-bean • wine • caesarian-section • occupancy-detection • vertebral-column •

Original: • wine

Original: • caesarian-section

Original: • occupancy-detection

Original: • vertebral-column

Original: • student-performance

Original: • shill-bidding

Original:  • mammographic-mass • teaching-assistant-evaluation • somerville-happiness • stroke-prediction• job-change• campus-placement• engineering-placement• water-potability• color-luminance• proteinogenic-acid
Revised: student-performance • shill-bidding • mammographic-mass • teaching-assistant-evaluation • somerville-happiness • stroke-prediction • job-change • campus-placement • engineering-placement • water-potability • color-luminance • proteinogenic-acid

Original: Novel tasks (16) for CLUES-Real are:
Revised: Novel tasks (16) for CLUES-Real are:

Original: • banknote-authentication• tic-tac-toe-endgame• car-evaluation• contraceptive-method-choice• indian-liver-patient• travel-insurance• entrepreneur-competency• award-nomination-result• coin-face-value• coin-metal• driving-championship-points• election-outcome• hotel-rating• manifold-orientability• soccer-club-region• soccer-league-type
Revised: • banknote-authentication • tic-tac-toe-endgame • car-evaluation • contraceptive-method-choice • indian-liver-patient • travel-insurance • entrepreneur-competency • award-nomination-result • coin-face-value • coin-metal • driving-championship-points • election-outcome • hotel-rating • manifold-orientability • soccer-club-region • soccer-league-type

Original: We train on 70% of the labeled examples ofthe seen tasks and perform zero-shot generalization test over the 20% examples of each task in
Revised: We train on 70% of the labeled examples of the seen tasks and perform zero-shot generalization test over the 20% examples of each task in

Original: CLUES-Real .
Revised: CLUES-Real .

Original: For the extremely small Wikipediatasks (for which we do not crowdsource explanations), we use the entire set of examples for zeroshot testing.
Revised: For the extremely small Wikipedia tasks (for which we do not crowdsource explanations), we use the entire set of examples for zeroshot testing.

Original: For CLUES-Synthetic , we have 96 tasks asseen (training) tasks and 48 as novel tasks.
Revised: For CLUES-Synthetic , we have 96 tasks as seen (training) tasks and 48 as novel tasks.

Original: Taskin CLUES-Synthetic that belong to the followingschemas are part of the seen tasks:
Revised: Task in CLUES-Synthetic that belong to the following schemas are part of the seen tasks:

Original: • Species of Animal• Species of Bird• Rainfall predictionTasks belonging to ‘Bond relevance classification’
Revised: • Species of Animal • Species of Bird • Rainfall prediction

Original:  and ‘League Rank Classification’ were part of novel tasks for CLUES-Synthetic .
Revised: Tasks belonging to ‘Bond relevance classification’ and ‘League Rank Classification’ were part of novel tasks for CLUES-Synthetic .

Original: We train on700 labeled examples of each seen task and performzero-shot generalization test over 200 examples of each novel task in CLUES-Synthetic .
Revised: We train on 700 labeled examples of each seen task and perform zero-shot generalization test over 200 examples of each novel task in CLUES-Synthetic .

Original: F. Model parameters
Revised: F. Model parameters

Original: • RoBERTa w/o Exp.
Revised: • RoBERTa w/o Exp.

Original: : The number of parameters is same as the pretrained RoBERTa-base modelavailable on HuggingFace library.
Revised: : The number of parameters is same as the pretrained RoBERTa-base model available on HuggingFace library.

Original: • RoBERTa w/ Exp.:
Revised: • RoBERTa w/ Exp. :

Original: The number of parameters is same as the pretrained RoBERTa-base model available on HuggingFace library.
Revised: The number of parameters is same as the pretrained RoBERTa-base model available on HuggingFace library.

Original: • ExEnt : We consider a pre-trained RoBERTa model fine-tuned on MNLI (Williams et al.,
Revised: • ExEnt : We consider a pre-trained RoBERTa model fine-tuned on MNLI (Williams et al.,

Original: 2017) corpus as our base entailment model.
Revised: 2017) corpus as our base entailment model.

Original: The number of parameters in ExEnt is same as the pre-trained RoBERTa mdoel finetuned on
Revised: The number of parameters in ExEnt is same as the pre-trained RoBERTa mdoel finetuned on

Original: MNLI corpus.
Revised: MNLI corpus.

Original: Further, in order to perform the

Original: assignment of logits using an explanation,

Original: we

Original:  maintain meta-information for each explanation to (1) determine if the explanation mentions to ‘assign’ a label or ‘not assign’ a label, and (2) track l exp (label mentioned in explanation).
Revised: Further, in order to perform the assignment of logits using an explanation, we maintain meta-information for each explanation to (1) determine if the explanation mentions to ‘assign’ a label or ‘not assign’ a label, and (2) track l exp (label mentioned in explanation).

Original: For
Revised: For

Original: CLUES-Synthetic , we parse the templated ex- planations to obtain the meta-information while for the explanations in CLUES-Real , the authors 
Revised: CLUES-Synthetic , we parse the templated explanations to obtain the meta-information, while for the explanations in CLUES-Real , the authors manually annotate this meta-information.

Original: manually annotate this meta-information.

Original: F. Hyper-parameter settings
Revised: F. Hyper-parameter settings

Original: For all the transformer based models we use the implementation of HuggingFace library (Wolf et al., 2020).
Revised: For all the transformer based models we use the implementation of HuggingFace library (Wolf et al., 2020).

Original: All the model based hyper-parameters are thus kept default to the settings in the HuggingFace library.
Revised: All the model based hyper-parameters are thus kept default to the settings in the HuggingFace library.

Original: We use the publicly available checkpoints to initialise the pre-trained models.
Revised: We use the publicly available checkpoints to initialise the pre-trained models.

Original: For RoBERTa based baselines we use ‘roberta-base’ checkpoint available on HuggingFace.
Revised: For RoBERTa based baselines we use ‘roberta-base’ checkpoint available on HuggingFace.

Original: For our intermediate entailment model in ExEnt , we finetune a pretrained checkpoint of RoBERTa trained on MNLI corpus (‘textattack/roberta-base-MNLI’)
Revised: For our intermediate entailment model in ExEnt , we finetune a pretrained checkpoint of RoBERTa trained on MNLI corpus (‘textattack/roberta-base-MNLI’)

Original: When training on CLUES-Synthetic , we use a maximum of 64 tokens for our baseline RoBERTa w/o Exp. and ExEnt .
Revised: When training on CLUES-Synthetic , we use a maximum of 64 tokens for our baseline RoBERTa w/o Exp. and ExEnt .

Original: For the RoBERTa w/ Exp.
Revised: For the RoBERTa w/ Exp.

Original: model we increase this limit to 128 tokens as it takes concatenation of all explanations for a task.
Revised: model we increase this limit to 128 tokens as it takes concatenation of all explanations for a task.

Original: When training on CLUES-Real , we use 256 tokens as limit for RoBERTa w/ Exp.
Revised: When training on CLUES-Real , we use 256 tokens as limit for RoBERTa w/ Exp.

Original: using explanations as the real-world tasks have roughly two times more explanations on average than synthetic tasks.
Revised: using explanations as the real-world tasks have roughly two times more explanations on average than synthetic tasks.

Original: We used the AdamW (Loshchilov and Hutter, 2019) optimizer commonly used to fine-tune pretrained Masked Language Models (MLM) models.
Revised: We used the AdamW (Loshchilov and Hutter, 2019) optimizer commonly used to fine-tune pretrained Masked Language Models (MLM) models.

Original: For fine-tuning the pre-trained models on our benchmark tasks, we experimented with learning rates { 1 e − 5 , 2 e − 5 } and chose 1 e − 5 based on performance on the performance on the validation set of seen tasks.
Revised: For fine-tuning the pre-trained models on our benchmark tasks, we experimented with learning rates { 1 e − 5 , 2 e − 5 } and chose 1 e − 5 based on performance on the performance on the validation set of seen tasks.

Original: Batch sizes was kept as 2 with gradient accumulation factor of 8.
Revised: Batch sizes was kept as 2 with gradient accumulation factor of 8.

Original: The random seed for all experiments was 42.
Revised: The random seed for all experiments was 42.

Original: We train all themodels for 20 epochs.
Revised: We train all the models for 20 epochs.

Original: Each epoch comprises of100 batches, and in each batch the models look atone of the tasks (in a sequential order) in the seensplit.
Revised: Each epoch comprises of 100 batches, and in each batch the models look at one of the tasks (in a sequential order) in the seen split.

Original: F. Hardware and software specifications
Revised: F. ExEnt Implementation Details

Original: All the models are coded using Pytorch 1.4.0 8(Paszke et al., 2019) and related libraries likenumpy (Harris et al., 2020), scipy (Jones et al.
Revised: In experiments, we consider a pre-trained RoBERTa model fine-tuned on MNLI (Williams et al., 2017) corpus as our base entailment model.

Revised: Further, in order to perform the assignment of logits using an explanation, we maintain metainformation for each explanation to (1) determine if the explanation mentions to ‘assign’ a label or ‘not assign’ a label, and (2) track l exp (label mentioned in explanation).

Original: ,2001–) etc.
Revised: For CLUES-Synthetic , we parse the templated explanations to obtain the meta-information, while for the explanations in CLUES-Real , the authors manually annotate this meta-information. F. Hardware and software specifications All the models are coded using Pytorch 1.4.0(Paszke et al., 2019) and related libraries like numpy (Harris et al., 2020), scipy (Jones et al., 2001–) etc.

Original: We run all experiments on one of thefollowing two systems - (1) GeForce RTX
Revised: We run all experiments on one of the following two systems - (1) GeForce RTX

Original: GPU of size 12 GB, 256 GB RAM and 40 CPUcores (2) Tesla V100-SXM2 GPU of size 16GB, 250 GB RAM and 40 CPU cores.
Revised: GPU of size 12 GB, 256 GB RAM and 40 CPU cores (2) Tesla V100-SXM2 GPU of size 16GB, 250 GB RAM and 40 CPU cores.

Original: F. Training times
Revised: F. Training times

Original: • Training on CLUES-Real : The baselineRoBERTa w/o Exp model typically takes 3 seconds on average for training on 1 batch of examples.
Revised: • Training on CLUES-Real : The baseline RoBERTa w/ o Exp model typically takes 3 seconds on average for training on 1 batch of examples.

Original: In 1 batch, the model goes through 16examples from the tasks in seen split.
Revised: In 1 batch, the model goes throughexamples from the tasks in seen split.

Original: RoBERTaw/ Exp. takes around 5 seconds to train on 1batch.
Revised: RoBERTa w/ Exp. takes around 5 seconds to train onbatch.

Original: ExEnt takes longer time than baselinesowing to the multiple forward passes.
Revised: ExEnt takes longer time than baselines owing to the multiple forward passes.

Original: For training on 1 batch of CLUES-Real , ExEnt took 12seconds on average.• Training on CLUES-Synthetic : All the modelstake comparatively much lesser time for training on our synthetic tasks owing to lesser number of explanations on average for a task.
Revised: For training on 1 batch of CLUES-Real , ExEnt took seconds on average. • Training on CLUES-Synthetic : All the models take comparatively much lesser time for training on our synthetic tasks owing to lesser number of explanations on average for a task.

Original: Fortraining on 1 batch, all models took 1 secondsor less to train on 1 batch of examples fromCLUES-Synthetic
Revised: For training on 1 batch, all models took 1 seconds or less to train on 1 batch of examples from CLUES-Synthetic

Original: G Key Challenges
Revised: G Key Challenges

Original: We identify the open challenges in learning fromexplanations by ablating the linguistic componentsand structure of explanations.
Revised: We identify the open challenges in learning from explanations by ablating the linguistic components [Figure] R e l a t i v e P e

Revised: r

Revised: f

Original:  For a robust analysis, we generate more tasks for each task type in
Revised: o r m a n c e G a i n ( [Figure] and structure of explanations. For a robust analysis, we generate more tasks for each task type in

Original: CLUES-Synthetic , making 100 tasks for each ofthe 48 different task-types in CLUES-Synthetic(axes of variation include 4 negation types, 3 conjunction/disjunction types, 2 quantifier types, andnumber of labels; details in Appendix B.5).
Revised: CLUES-Synthetic , making 100 tasks for each of the 48 different task-types in CLUES-Synthetic (axes of variation include 4 negation types, 3 conjunction/disjunction types, 2 quantifier types, and number of labels; details in Appendix B.5).

Original: We evaluate the generalization performance of
Revised: We evaluate the generalization performance of

Original: ExEnt to novel tasks on each of the different typesseparately by training separate models for eachtask type.
Revised: ExEnt to novel tasks on each of the different types separately by training separate models for each task type.

Original: Figure 10 shows the relative gain in generalization performance of models learned usingorg explanations compared to the performance of baseline RoBERTa w/o Exp.
Revised: Figure 10 shows the relative gain in generalization performance of models learned using explanations compared to the performance of baseline RoBERTa w/o Exp.

Original: Our results indicate that learning from explanations containing quantifiers is highly challenging.
Revised: Our results indicate that learning from explanations containing quantifiers is highly challenging.

Original: In the presence of quantifiers, models guided by explanations perform on par with the baseline RoBERTa w/o Exp model.
Revised: In the presence of quantifiers, models guided by explanations perform on par with the baseline RoBERTa w/o Exp model.

Original: Negations also pose a challenge, as indicated by the decline in relative gains of models guided by explanation compared to the RoBERTa w/o Exp model.
Revised: Negations also pose a challenge, as indicated by the decline in relative gains of models guided by explanation compared to the RoBERTa w/o Exp model.

Original: Structurally complex explanations (containing conjunctions/disjunctions of clauses) are also hard to learn from compared to simple conditional statements.
Revised: Structurally complex explanations (containing conjunctions/disjunctions of clauses) are also hard to learn from compared to simple conditional statements.

Original: These challenges provide a fertile ground for exciting future research.
Revised: These challenges provide a fertile ground for exciting future research.

Original: H Annotation interfaces
Revised: H Annotation interfaces

Original: We present the different annotation templates and interfaces used for our explanation collection and verification stages in Figures 11,12,13,14 and Figure 15 respectively. 
Revised: We present the different annotation templates and interfaces used for our explanation collection and verification stages in Figures 11,12,13,14 and Figure 15 respectively. D ATASET

Original: [Figure]
Revised: [Figure]

Original: l a t i v e

Original: P e r f
Revised: [Figure]

Original: o r m a n c e G
Revised: authentication

Original: [Figure]
Revised: [Figure]

Original: i
Revised: choice

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Table] 
Revised: [Table] [Figure]

Original: [Table] [Table] [Table]
Revised: [Table] [Figure]

Original: [Table] [Table]
Revised: [Table] [Figure]

Revised: caesarian-

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: This

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table][Table]
Revised: [Table] [Table]

Original: [Table] [Table][Table]
Revised: [Table] [Table]

Original: [Table] [Table][Table]
Revised: [Table] [Table]