Original: CLUES : A Benchmark for Learning Classifiers using Natural Language Explanations
Revised: CLUES : A Benchmark for Learning Classifiers using Natural Language Explanations

Original: Abstract
Revised: Abstract

Original: Supervised learning has traditionally focused on inductive learning by observing labeled examples of a task.
Revised: Supervised learning has traditionally focused on inductive learning by observing labeled examples of a task.

Original: In contrast, humans have the ability to learn new concepts from language.
Revised: In contrast, humans have the ability to learn new concepts from language.

Original: Here, we explore training classifiers for structured data 1 purely from language.
Revised: Here, we explore training classifiers for structured data 1 purely from language.

Original: We introduce
Revised: We introduce

Original: CLUES , a benchmark for C lassifier L earning
Revised: CLUES , a benchmark for C lassifier L earning

Original: U sing natural language E xplanation S , consisting of a range of classification tasks over structured data along with language supervision in the form of explanations.
Revised: U sing natural language E xplanation S , consisting of a range of classification tasks over structured data along with language supervision in the form of explanations.

Original: CLUES consists of 36 real-world and 144 synthetic classification tasks.
Revised: CLUES consists of 36 real-world and 144 synthetic classification tasks.

Original: It contains crowdsourced explanations describing real-world tasks from multiple teachers and programmatically generated explanations for the synthetic tasks.
Revised: It contains crowdsourced explanations describing real-world tasks from multiple teachers and programmatically generated explanations for the synthetic tasks.

Original: We develop
Revised: We develop

Original: ExEnt , an entailment-based model that learns classifiers using explanations.
Revised: ExEnt , an entailment-based model that learns classifiers using explanations.

Original: ExEnt generalizes up to 18% better (relative) on novel tasks than a baseline that does not use explanations.
Revised: ExEnt generalizes up to 18% better (relative) on novel tasks than a baseline that does not use explanations.

Original: Our code and datasets are available at: https: //clues-benchmark.github.io .
Revised: Our code and datasets are available at: https: //clues-benchmark.github.io .

Original:  Introduction
Revised:  Introduction

Original: Humans have a remarkable ability to learn concepts through language (Chopra et al., 2019; Tomasello, 1999).
Revised: Humans have a remarkable ability to learn concepts through language (Chopra et al., 2019; Tomasello, 1999).

Original: For example, we can learn about poisonous mushrooms through an explanation like ‘a mushroom is poisonous if it has pungent odor’ .
Revised: For example, we can learn about poisonous mushrooms through an explanation like ‘a mushroom is poisonous if it has pungent odor’ .

Original: Such an approach profoundly contrasts with the predominant paradigm of machine learning, where algorithms extract patterns by looking at scores of labeled examples of poisonous and edible mushrooms.
Revised: Such an approach profoundly contrasts with the predominant paradigm of machine learning, where algorithms extract patterns by looking at scores of labeled examples of poisonous and edible mushrooms.

Original: However, it is unnatural to presume the availability of labeled examples for the heavy tail of naturally occurring concepts in the world.
Revised: However, it is unnatural to presume the availability of labeled examples for the heavy tail of naturally occurring concepts in the world.

Original: This work studies how models trained to learn from natural language explanations can general-
Revised: This work studies how models trained to learn from natural language explanations can general-

Original: [Table]
Revised: [Table]

Original: ize to novel tasks without access to labeled examples.
Revised: ize to novel tasks without access to labeled examples.

Original: While prior works in this area (Srivastava et al., 2017, 2018; Hancock et al., 2018; Murty et al., 2020, inter alia ) have explored explanations as a source of supervision, they evaluate models on a small number of tasks (2-3 relation extraction tasks in Hancock et al.
Revised: While prior works in this area (Srivastava et al., 2017, 2018; Hancock et al., 2018; Murty et al., 2020, inter alia ) have explored explanations as a source of supervision, they evaluate models on a small number of tasks (2-3 relation extraction tasks in Hancock et al.

Original: ; Murty et al.
Revised: ; Murty et al.

Original: (2020), 7 email categorization tasks (Srivastava et al., 2017)).
Revised: (2020), 7 email categorization tasks (Srivastava et al., 2017)).

Original: Owing to the paucity of largescale benchmarks for learning from explanations over diverse tasks, we develop CLUES , a benchmark of classification tasks paired with natural language explanations.
Revised: Owing to the paucity of largescale benchmarks for learning from explanations over diverse tasks, we develop CLUES , a benchmark of classification tasks paired with natural language explanations.

Original: Our benchmark is divided into CLUES-Real and CLUES-Synthetic consisting of tasks from real-world (UCI, Kaggle, and
Revised: Our benchmark is divided into CLUES-Real and CLUES-Synthetic consisting of tasks from real-world (UCI, Kaggle, and

Original: Wikipedia) and synthetic domains respectively.
Revised: Wikipedia) and synthetic domains respectively.

Original: Explanations for CLUES-Real are crowdsourced to mimic the diversity and difficulty of human learning and pedagogy.
Revised: Explanations for CLUES-Real are crowdsourced to mimic the diversity and difficulty of human learning and pedagogy.

Original: For CLUES-Synthetic , we generate the explanations programmatically to test
Revised: For CLUES-Synthetic , we generate the explanations programmatically to test

Original: [Figure]
Revised: [Figure]

Original: models’ reasoning ability under a range of structural and linguistic modifications of explanations.
Revised: models’ reasoning ability under a range of structural and linguistic modifications of explanations.

Original: In addition to creating CLUES , we train models with a mix of explanations and labeled examples, in a multi-task setup, over a set of seen classification tasks to induce generalization to novel tasks, where we do not have any labeled examples.
Revised: In addition to creating CLUES , we train models with a mix of explanations and labeled examples, in a multi-task setup, over a set of seen classification tasks to induce generalization to novel tasks, where we do not have any labeled examples.

Original: However, we notice that simply concatenating explanations to the input does not help pre-trained models, like
Revised: However, we notice that simply concatenating explanations to the input does not help pre-trained models, like

Original: RoBERTa (Liu et al., 2019), generalize.
Revised: RoBERTa (Liu et al., 2019), generalize.

Original: Hence, we develop ExEnt , an entailment-based model for learning classifiers guided by explanations.
Revised: Hence, we develop ExEnt , an entailment-based model for learning classifiers guided by explanations.

Original: ExEnt shows a relative improvement of up to 18% over other baselines on novel tasks.
Revised: ExEnt shows a relative improvement of up to 18% over other baselines on novel tasks.

Original: Our contributions are:
Revised: Our contributions are:

Original: • We introduce CLUES , a benchmark for learning classifiers over structured data from language.
Revised: • We introduce CLUES , a benchmark for learning classifiers over structured data from language.

Original: • We develop ExEnt , an entailment-based model for learning classifiers guided by explanations.
Revised: • We develop ExEnt , an entailment-based model for learning classifiers guided by explanations.

Original:  Creating CLUES
Revised:  Creating CLUES

Original: In CLUES , we frame classification tasks over structured data represented in tabular format.
Revised: In CLUES , we frame classification tasks over structured data represented in tabular format.

Original: We consider two splits of our benchmark, CLUES-Real (real-world datasets) and CLUES-Synthetic (synthetic datasets).
Revised: We consider two splits of our benchmark, CLUES-Real (real-world datasets) and CLUES-Synthetic (synthetic datasets).

Original: 2. CLUES-Real
Revised: 2. CLUES-Real

Original: We first gather/create classification tasks from UCI,
Revised: We first gather/create classification tasks from UCI,

Original: Kaggle, and Wikipedia tables, then collect explanations for each classification task.
Revised: Kaggle, and Wikipedia tables, then collect explanations for each classification task.

Original: Collecting datasets: We choose 18 tabular classification tasks from UCI ML repository 2 andfrom Kaggle 3 .
Revised: Collecting datasets: We choose 18 tabular classification tasks from UCI ML repository 2 andfrom Kaggle 3 .

Original: Additionally, we mine suitable tables from Wikipedia that can be posed as classification tasks.
Revised: Additionally, we mine suitable tables from Wikipedia that can be posed as classification tasks.

Original: We formalize the mining process as a crowdsourcing task (details in Appendix C.2).
Revised: We formalize the mining process as a crowdsourcing task (details in Appendix B.2).

Original: We identified 11 classification tasks corresponding to 9 Wikipedia tables after mining around 10K Wikipedia tables.
Revised: We identified 11 classification tasks corresponding to 9 Wikipedia tables after mining around 10K Wikipedia tables.

Original: The details of tasks in
Revised: The details of tasks in

Original: CLUES-Real are provided in Appendix C.
Revised: CLUES-Real are provided in Appendix B.

Original: Explanation Collection 
Revised: Explanation Collection Pipeline: 1. C OLLECTING EXPLANATIONS :

Original: Pipeline : 1. C OLLECTING EXPLANATIONS :

Original: We use the
Revised: We use the

Original: Amazon Mechanical Turk (AMT) platform to collect explanations for CLUES-Real .
Revised: Amazon Mechanical Turk (AMT) platform to collect explanations for CLUES-Real .

Original: In each
Revised: In each

Original: HIT, we provide turkers with a few labeled examples of a task and ask them to provide a set of explanations corresponding to the task.
Revised: HIT, we provide turkers with a few labeled examples of a task and ask them to provide a set of explanations corresponding to the task.

Original: The turkers participating in this task have been vetted by a qualification task to test their understanding of ‘good’ and ‘bad’ explanations (see
Revised: The turkers participating in this task have been vetted by a qualification task to test their understanding of ‘good’ and ‘bad’ explanations (see

Original: Appendix H for templates).
Revised: Appendix G for templates).

Original: After providing explanations, the turkers predict classes for a set of unlabeled samples by using their explanations.
Revised: After providing explanations, the turkers predict classes for a set of unlabeled samples by using their explanations.

Original: The turkers in this stage are henceforth referred as ‘teachers’ in our setup since they provide explanations to ‘teach’ models about different classification tasks.
Revised: The turkers in this stage are henceforth referred as ‘teachers’ in our setup since they provide explanations to ‘teach’ models about different classification tasks.

Original: 2. E XPLANATION VERIFICATION
Revised: 2. E XPLANATION VERIFICATION

Original: We validate the utility of the explanations for a task from each teacher by evaluating if they are useful for other humans in learning the task.
Revised: We validate the utility of the explanations for a task from each teacher by evaluating if they are useful for other humans in learning the task.

Original: We perform explanation verification on AMT as well, where we ask a new set of turkers to label a set of unlabeled samples from a task using the explanations provided by individual teachers in the ‘explanation collection’ phase.
Revised: We perform explanation verification on AMT as well, where we ask a new set of turkers to label a set of unlabeled samples from a task using the explanations provided by individual teachers in the ‘explanation collection’ phase.

Original: Additionally, we ask the turkers to give a Likert rating (1-4 scale) on the usefulness of each explanation.
Revised: Additionally, we ask the turkers to give a Likert rating (1-4 scale) on the usefulness of each explanation.

Original: Since the turkers in the verification stage perform the classification task using language explanations from a teacher, we refer to them as ‘students’ for our setup henceforth.
Revised: Since the turkers in the verification stage perform the classification task using language explanations from a teacher, we refer to them as ‘students’ for our setup henceforth.

Original: [Figure]
Revised: [Figure]

Original: equen cy
Revised: equen cy

Original: Likert Ratings of Explanations
Revised: Likert Ratings of Explanations

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: 2. CLUES-Synthetic
Revised: 2. CLUES-Synthetic

Original: To delineate challenges in learning from explanations under controlled settings, we create
Revised: To delineate challenges in learning from explanations under controlled settings, we create

Original: CLUES-Synthetic , a set of programmatically created classification tasks with varying complexity of explanations (in terms of structure and presence of quantifiers, conjunctions, etc.)
Revised: CLUES-Synthetic , a set of programmatically created classification tasks with varying complexity of explanations (in terms of structure and presence of quantifiers, conjunctions, etc.)

Original: and concept definitions.
Revised: and concept definitions.

Original: We use the following types of rules that differ in structure and complexity ( c i denotes i th clause and l denotes a label):
Revised: We use the following types of rules that differ in structure and complexity ( c i denotes i th clause and l denotes a label):

Original: Simple:
Revised: Simple:

Original: IF c 1 THEN l • Conjunctive: IF c 1 AND c 2 THEN l • Disjunctive: IF c 1 OR c 2 THEN l • Nested disjunction over conjunction: IF c 1 OR ( c 2 AND c 3 ) THEN l • Nested conjunction over disjunction: IF cAND ( c 2 OR c 3 ) THEN l • For each of the above, we include variants with negations (in clauses and/or labels): for example, IF c 1 THEN NOT l
Revised: IF c 1 THEN l • Conjunctive: IF c 1 AND c 2 THEN l • Disjunctive: IF c 1 OR c 2 THEN l • Nested disjunction over conjunction: IF c 1 OR ( c 2 AND c 3 ) THEN l • Nested conjunction over disjunction: IF cAND ( c 2 OR c 3 ) THEN l • For each of the above, we include variants with negations (in clauses and/or labels): for example, IF c 1 THEN NOT l

Original: We also consider other linguistic variations of rules by inserting quantifiers (such as ‘always’, ‘likely’).
Revised: We also consider other linguistic variations of rules by inserting quantifiers (such as ‘always’, ‘likely’).

Original: The synthetic explanations are template-generated from the rules used in creating the task.
Revised: The synthetic explanations are template-generated from the rules used in creating the task.

Original: For brevity, we defer additional details on the use of quantifiers, label assignment using rules, and creation of synthetic explanations to Appendix B. Overall we have 48 different task types (based on the number of classes and rule variants) using which we synthetically create 144 classification tasks.
Revised: For brevity, we defer additional details on the use of quantifiers, label assignment using rules, and creation of synthetic explanations to Appendix A. Overall we have 48 different task types (based on the number of classes and rule variants) using which we synthetically create 144 classification tasks.

Original:  Dataset analysis
Revised:  Dataset analysis

Original: In this section, we analyze the tasks and the collected explanations in CLUES .
Revised: In this section, we analyze the tasks and the collected explanations in CLUES .

Original: Task Statistics :
Revised: Task Statistics :

Original: Table 1 shows the statistics of tasks in CLUES .
Revised: Table 1 shows the statistics of tasks in CLUES .

Original: Task details are provided in Appendices B and C. An aggregate of 133 teachers provide 318 explanations for tasks in CLUES-Real .
Revised: Task details are provided in Appendices A and B. An aggregate of 133 teachers provide 318 explanations for tasks in CLUES-Real .

Original: All collected explanations were manually filtered and irrelevant explanations were removed.
Revised: All collected explanations were manually filtered and irrelevant explanations were removed.

Original: Further, each explanation set corresponding to a teacher in
Revised: Further, each explanation set corresponding to a teacher in

Original: CLUES-Real was verified by 3 students.
Revised: CLUES-Real was verified by 3 students.

Original: Lexical analysis of explanations : Using the spacy tokenizer, we find the vocabulary size of explanations in CLUES as 1026 tokens resulting in 15.53 tokens on average per explanation.
Revised: Lexical analysis of explanations : Using the spacy tokenizer, we find the vocabulary size of explanations in CLUES as 1026 tokens resulting in 15.53 tokens on average per explanation.

Original: The median reading complexity of the explanations is 65.73 (8th/9th-grade reading level) 5 .
Revised: The median reading complexity of the explanations is 65.73 (8th/9th-grade reading level) 5 .

Original: Usefulness of the explanations : During explanation verification, turkers provide a rating (on a
Revised: Usefulness of the explanations : During explanation verification, turkers provide a rating (on a

Original: Likert scale from 1 to 4) on the utility of the explanations for classification (1 - ‘not helpful’ ; 4 - ‘denotes mostly helpful in prediction’).
Revised: Likert scale from 1 to 4) on the utility of the explanations for classification (1 - ‘not helpful’ ; 4 - ‘denotes mostly helpful in prediction’).

Original: The average rating for the explanations in CLUES-Real is 2.78, denoting most explanations were useful.
Revised: The average rating for the explanations in CLUES-Real is 2.78, denoting most explanations were useful.

Original: Characteristics of teachers and students : Figure 3(b) shows the normalized teacher performance vs normalized student performance for teacherstudent pairs in CLUES-Real .
Revised: Characteristics of teachers and students : Figure 3(b) shows the normalized teacher performance vs normalized student performance for teacherstudent pairs in CLUES-Real .

Original: Normalized performance of an individual teacher (or, student) on a task is defined as the difference between the performances of the teacher (or, student) and an average teacher (or, student) for the same task.
Revised: Normalized performance of an individual teacher (or, student) on a task is defined as the difference between the performances of the teacher (or, student) and an average teacher (or, student) for the same task.

Original: The positive correlation ( ρ = 0.17) suggests that students tend
Revised: The positive correlation ( ρ = 0.17) suggests that students tend

Original: [Figure]
Revised: [Figure]

Original: teacherstudentrandom to perform well if taught by well-performing teachers.
Revised: teacherstudentrandom to perform well if taught by well-performing teachers.

Original: Figure 4 shows average teacher and avergage student performance for each task in CLUES-Real .
Revised: Figure 4 shows average teacher and avergage student performance for each task in CLUES-Real .

Original: We find that an average teacher performs better than the average student on most tasks, barring a few tasks such as ‘tic-tac-toe’, which could be solved using commonsense without relying on explanations.
Revised: We find that an average teacher performs better than the average student on most tasks, barring a few tasks such as ‘tic-tac-toe’, which could be solved using commonsense without relying on explanations.

Original: Please refer to Appendix D for additional data analysis and insights on CLUES .
Revised: Please refer to Appendix C for additional data analysis and insights on CLUES .

Original:  Experiment Setup and Models
Revised:  Experiment Setup and Models

Original: In this section, we describe our training and evaluation setup, our models, and experimental findings.
Revised: In this section, we describe our training and evaluation setup, our models, and experimental findings.

Original: 4. Training and Evaluation Setup
Revised: 4. Training and Evaluation Setup

Original: Our goal is to learn a model that, at inference, can perform classification over an input x to obtain the class label y , given the set of explanations E for the classification task.
Revised: Our goal is to learn a model that, at inference, can perform classification over an input x to obtain the class label y , given the set of explanations E for the classification task.

Original: We train our model using multitask training over a set of tasks T seen and evaluate generalization to a new task, t ∈ T novel .
Revised: We train our model using multitask training over a set of tasks T seen and evaluate generalization to a new task, t ∈ T novel .

Original: The task split we use for our experiments can be found in
Revised: The task split we use for our experiments can be found in

Original: Appendix F.1.
Revised: Appendix E.1.

Original: We select our best model for zeroshot evaluation based on the validation scores on the seen tasks.
Revised: We select our best model for zeroshot evaluation based on the validation scores on the seen tasks.

Original: Following linearization techniques for structured data Yin et al.
Revised: Following linearization techniques for structured data Yin et al.

Original: (2020), we adapt a similar encoding process wherein we encode each structured data example, x , as a text sequence, by linearizing it as a sequence of attribute-name and
Revised: (2020), we adapt a similar encoding process wherein we encode each structured data example, x , as a text sequence, by linearizing it as a sequence of attribute-name and

Original: attribute-value
Revised: attribute-value

Original: pairs separated by [SEP] tokens.
Revised: pairs separated by [SEP] tokens.

Original: We will refer to the linearized format of structured inputs by ‘Features-as-Text’ or ‘FaT’.
Revised: We will refer to the linearized format of structured inputs by ‘Features-as-Text’ or ‘FaT’.

Original: 4. Baseline models
Revised: 4. Baseline models

Original: We consider the following baselines:
Revised: We consider the following baselines:

Original: • RoBERTa w/o Exp.
Revised: • RoBERTa w/o Exp.

Original: : This is a RoBERTabased (Liu et al., 2019) baseline that does not utilize the explanations to train a classifier.
Revised: : This is a RoBERTabased (Liu et al., 2019) baseline that does not utilize the explanations to train a classifier.

Original: The pre-trained RoBERTa model takes the linearized structured data (FaT) as input and outputs a representation for this context (using the [CLS] token).
Revised: The pre-trained RoBERTa model takes the linearized structured data (FaT) as input and outputs a representation for this context (using the [CLS] token).

Original: Next, we run another forward pass using RoBERTa to obtain a representation of the labels based on their text.
Revised: Next, we run another forward pass using RoBERTa to obtain a representation of the labels based on their text.

Original: We compute the probability distribution over labels by doing a dot-product of the representations of the input and the labels.
Revised: We compute the probability distribution over labels by doing a dot-product of the representations of the input and the labels.

Original: We train this model using cross-entropy loss.
Revised: We train this model using cross-entropy loss.

Original: • RoBERTa w/ Exp.
Revised: • RoBERTa w/ Exp.

Original: : This follows the same training setup as RoBERTa w/o Exp.
Revised: : This follows the same training setup as RoBERTa w/o Exp.

Original: However, in addition to FaT, this model also takes a concatenation of explanations as the input to RoBERTa.
Revised: However, in addition to FaT, this model also takes a concatenation of explanations as the input to RoBERTa.

Original: 4. ExEnt
Revised: 4. ExEnt

Original: We empirically notice that simply concatenating explanations to the input does not help pre-trained models, like RoBERTa generalize.
Revised: We empirically notice that simply concatenating explanations to the input does not help pre-trained models, like RoBERTa generalize.

Original: In order to better model the influence of an explanation towards deciding a class label, we draw analogies with the entailment of an explanation ( hypothesis ) towards the structured input ( premise ).
Revised: In order to better model the influence of an explanation towards deciding a class label, we draw analogies with the entailment of an explanation ( hypothesis ) towards the structured input ( premise ).

Original: Figure 5 shows the overview of our explanation-guided classification model, ExEnt .
Revised: Figure 5 shows the overview of our explanation-guided classification model, ExEnt .

Original: Given a structured input and explanation of a task, let l exp denote the label mentioned in the explanation, and L denote the set of labels of the task.
Revised: Given a structured input and explanation of a task, let l exp denote the label mentioned in the explanation, and L denote the set of labels of the task.

Original: The entailment model provides entailment ( p e ), contradiction ( p c ) and neutral ( p n ) logits which are then assigned to the class labels as:
Revised: The entailment model provides entailment ( p e ), contradiction ( p c ) and neutral ( p n ) logits which are then assigned to the class labels as:

Original: • If explanation mentions to assign a label :
Revised: • If explanation mentions to assign a label :

Original: Assign p e to l exp , p c is divided equally among labels in L \ { l exp } , and p n is divided equally among labels in L .
Revised: Assign p e to l exp , p c is divided equally among labels in L \ { l exp } , and p n is divided equally among labels in L .

Original: • If explanation mentions to not assign a label : This occurs if a negation is associated with l exp .
Revised: • If explanation mentions to not assign a label : This occurs if a negation is associated with l exp .

Original: Assign p c to l exp
Revised: Assign p c to l exp

Original: , p e is divided equally among labels in L \ { l exp } , and p n is divided equally among labels in L .
Revised: , p e is divided equally among labels in L \ { l exp } , and p n is divided equally among labels in L .

Original: We obtain logit scores over labels of the task corresponding to each explanation as described above.
Revised: We obtain logit scores over labels of the task corresponding to each explanation as described above.

Original: We compute the final label logits by aggregating (using mean) over the label logits corresponding to each explanation of the task.
Revised: We compute the final label logits by aggregating (using mean) over the label logits corresponding to each explanation of the task.

Original: The final label logits are converted to a probability distribution over labels, and we train ExEnt using cross-entropy loss.
Revised: The final label logits are converted to a probability distribution over labels, and we train ExEnt using cross-entropy loss.

Original: Additional training details and hyperparameters
Revised: Additional training details and hyperparameters

Original: MLM
Revised: MLM

Original: Exp.
Revised: Exp.

Original: FaT [SEP] [SEP] [CLS]
Revised: FaT [SEP] [SEP] [CLS]

Original: Exp.
Revised: Exp.

Original: FaT [SEP] [SEP] [CLS]
Revised: FaT [SEP] [SEP] [CLS]

Original: Exp.
Revised: Exp.

Original: FaT [SEP] [SEP] [CLS]
Revised: FaT [SEP] [SEP] [CLS]

Original: Entailment Logits
Revised: Entailment Logits

Original: Agg.
Revised: Agg.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: 4. Zero-Shot Generalization Performance
Revised: 4. Zero-Shot Generalization Performance

Original: Figure 6 shows the generalization performance of all models on CLUES .
Revised: Figure 6 shows the generalization performance of all models on CLUES .

Original: ExEnt outperforms the baselines suggesting that performing entailment as an intermediate step helps aggregate information from multiple explanations better.
Revised: ExEnt outperforms the baselines suggesting that performing entailment as an intermediate step helps aggregate information from multiple explanations better.

Original: On CLUES-Real ,
Revised: On CLUES-Real ,

Original: ExEnt gets an 18% relative improvement over the baselines while having an 11% relative improvement on CLUES-Synthetic .
Revised: ExEnt gets an 18% relative improvement over the baselines while having an 11% relative improvement on CLUES-Synthetic .

Original: To evaluate the utility of our synthetic tasks in enabling transfer learning to real-world tasks, we fine-tune a ExEnt model pre-trained on synthetic tasks.
Revised: To evaluate the utility of our synthetic tasks in enabling transfer learning to real-world tasks, we fine-tune a ExEnt model pre-trained on synthetic tasks.

Original: We experiment with three pre-training task sets - CLUES-Synthetic ,
Revised: We experiment with three pre-training task sets - CLUES-Synthetic ,

Original: CLUES-Synthetic (3x) and CLUES-Synthetic (5x) consisting of 144, 432, and 720 tasks.
Revised: CLUES-Synthetic (3x) and CLUES-Synthetic (5x) consisting of 144, 432, and 720 tasks.

Original: These larger synthetic task sets are created by sampling tasks from each of the 48 different synthetic tasks types similar to how CLUES-Synthetic was created (see §2.2 for reference).
Revised: These larger synthetic task sets are created by sampling tasks from each of the 48 different synthetic tasks types similar to how CLUES-Synthetic was created (see §2.2 for reference).

Original: We find that pretraining on synthetic tasks boosts the performance of ExEnt on the novel tasks of CLUES-Real by up to 39% (relative) over the RoBERTa w/o Exp. model.
Revised: We find that pretraining on synthetic tasks boosts the performance of ExEnt on the novel tasks of CLUES-Real by up to 39% (relative) over the RoBERTa w/o Exp. model.

Original: Additional experiments on synthetic tasks also reveal that ExEnt struggles with handling negations, conjunctions and disjunctions when learning from explanations (details in Appendix G).
Revised: Additional experiments on synthetic tasks also reveal that ExEnt struggles with handling negations, conjunctions and disjunctions when learning from explanations (details in Appendix F).

Original: Human Performance
Revised: Human Performance

Original: To situate the performance of the automated models, we performed human evaluation for tasks in test split of
Revised: To situate the performance of the automated models, we performed human evaluation for tasks in test split of

Original: CLUES-Real using AMT.
Revised: CLUES-Real using AMT.

Original: For this, we sampled at most 50 examples 6 from the test split of tasks in
Revised: For this, we sampled at most 50 examples 6 from the test split of tasks in

Original: CLUES-Real and each example was ‘labeled’ byturkers using the explanations of the ‘best teacher’ (the teacher whose students got the best performance during ‘explanation verification’ stage; see §2 for reference).
Revised: CLUES-Real and each example was ‘labeled’ byturkers using the explanations of the ‘best teacher’ (the teacher whose students got the best performance during ‘explanation verification’ stage; see §2 for reference).

Original: The average human accuracy for this was about 70%.
Revised: The average human accuracy for this was about 70%.

Original: However, the performance numbers of humans and models are not directly comparable as the model looks at all the explanations for the task, whereas the humans observe a small number of explanations.
Revised: However, the performance numbers of humans and models are not directly comparable as the model looks at all the explanations for the task, whereas the humans observe a small number of explanations.

Original: Humans also see multiple examples of the task during the evaluation, which they can use to fine-tune their understanding of a concept.
Revised: Humans also see multiple examples of the task during the evaluation, which they can use to fine-tune their understanding of a concept.

Original: The automated models don’t have a mechanism to leverage such data.
Revised: The automated models don’t have a mechanism to leverage such data.

Original:  Conclusion
Revised:  Conclusion

Original: We introduce CLUES , a benchmark with diverse classification tasks over structured data using natural language explanations to test the ability of models to learn novel classification tasks purely from language.
Revised: We introduce CLUES , a benchmark with diverse classification tasks over structured data using natural language explanations to test the ability of models to learn novel classification tasks purely from language.

Original: Additionally, we introduce ExEnt , an entailment-based model to learn classifiers guided by explanations.
Revised: Additionally, we introduce ExEnt , an entailment-based model to learn classifiers guided by explanations.

Original: Our results indicate that explicitly modeling the role of each explanation through entailment can enable learning classifiers for new tasks from explanations.
Revised: Our results indicate that explicitly modeling the role of each explanation through entailment can enable learning classifiers for new tasks from explanations.

Original: Future work can explore open challenges such as modeling quantifiers and negations present in an explanation.
Revised: Future work can explore open challenges such as modeling quantifiers and negations present in an explanation.

Original: CLUES is agnostic in the domain of tasks allowing the research community to contribute more tasks in the future.
Revised: CLUES is agnostic in the domain of tasks allowing the research community to contribute more tasks in the future.

Original: A Code and Models

Original: Code and models for CLUES are submitted as supplementary material and will be publicly released upon first publication.

Original: B Additional details on creating CLUES-Synthetic
Revised: A Additional details on creating CLUES-Synthetic

Original: In this section we discuss in detail about the various table schemas followed by the details of quantifiers and label assignment for creating synthetic tasks.
Revised: In this section we discuss in detail about the various table schemas followed by the details of quantifiers and label assignment for creating synthetic tasks.

Original: B. Tables schemas
Revised: A. Tables schemas

Original: We define five different table schemas, each corresponding to a different domain
Revised: We define five different table schemas, each corresponding to a different domain.

Original: For all the attributes in a schema we define a fixed domain from which values 
Revised: For all the attributes in a schema we define a fixed domain from which values for that attribute can be sampled.

Original: for that attribute can be sampled • Species of bird : The classification task here is to classify a bird into a particular species based on various attributes (column names in table).
Revised: • Species of bird : The classification task here is to classify a bird into a particular species based on various attributes (column names in table).

Original: We define several artificial species of birds using commonly used nonce words in psychological studies (Chopra et al., 2019) such as “dax", “wug", etc.
Revised: We define several artificial species of birds using commonly used nonce words in psychological studies (Chopra et al., 2019) such as “dax", “wug", etc.

Original: • Species of animal :
Revised: • Species of animal :

Original: The classification task here is to classify an animal into a particular species based on various attributes (column names in table).
Revised: The classification task here is to classify an animal into a particular species based on various attributes (column names in table).

Original: Artificial species of animals are again defined using commonly used nonce words in psychological studies such as “dax", “wug", etc.
Revised: Artificial species of animals are again defined using commonly used nonce words in psychological studies such as “dax", “wug", etc.

Original: • Rainfall prediction : This is a binary classification task where the objective is to predict whether it will rain tomorrow based on attributes such as “location", “minimum temperature", “humidity", “atmospheric pressure" etc.
Revised: • Rainfall prediction : This is a binary classification task where the objective is to predict whether it will rain tomorrow based on attributes such as “location", “minimum temperature", “humidity", “atmospheric pressure" etc.

Original: • Rank in league : This is a multi-label classification task where given attributes such “win percentage", “power rating", “field goal rating" of a basketball club, the objective is to predict its position in the league out of 1, 2, 3, 4, "Not qualified".
Revised: • Rank in league : This is a multi-label classification task where given attributes such “win percentage", “power rating", “field goal rating" of a basketball club, the objective is to predict its position in the league out of 1, 2, 3, 4, "Not qualified".

Original: • Bond relevance : This is a multi-label classification task where given attributes such “user age", “user knowledge", “user income", the objective is to predict the relevance of a bond out of 5 classes (1 to 5).
Revised: • Bond relevance : This is a multi-label classification task where given attributes such “user age", “user knowledge", “user income", the objective is to predict the relevance of a bond out of 5 classes (1 to 5).

Original: In each of the above schemas, the attributes can be either of types categorical or numeral.
Revised: In each of the above schemas, the attributes can be either of types categorical or numeral.

Original: For each of the above schemas we also define range of admissible values for each attribute.
Revised: For each of the above schemas we also define range of admissible values for each attribute.

Original: Detailed description of schemas are provided in Tables 5, 6, 7 8, 9.
Revised: Detailed description of schemas are provided in Tables 5, 6, 7 8, 9.

Original: B. List of quantifiers
Revised: A. List of quantifiers

Original: The full list of quantifiers along with their associated probability values are shown in Table 2.
Revised: The full list of quantifiers along with their associated probability values are shown in Table 2.

Original: [Table]
Revised: [Table]

Original: B. Creating synthetic explanations
Revised: A. Creating synthetic explanations

Original: We use a template-based approach to convert the set to rules into language explanations.
Revised: We use a template-based approach to convert the set to rules into language explanations.

Original: We convert every operator in the clauses into their corresponding language format as:
Revised: We convert every operator in the clauses into their corresponding language format as:

Original: • ==
Revised: • ==

Original: → ‘equal to’ • > → ‘greater than’ • > =
Revised: → ‘equal to’ • > → ‘greater than’ • > =

Original: → ‘greater than or equal to’ • < → ‘lesser than’ • < = → ‘lesser than or equal to’ • !
Revised: → ‘greater than or equal to’ • < → ‘lesser than’ • < = → ‘lesser than or equal to’ • !

Original: =
Revised: =

Original: → "not equal to’ • ! >
Revised: → "not equal to’ • ! >

Original: → ‘not greater than’ • !
Revised: → ‘not greater than’ • !

Original: < → ‘not lesser than’
Revised: < → ‘not lesser than’

Original: For example if we have a rule ‘ IF number of hands =
Revised: For example if we have a rule ‘ IF number of hands =

Original: = 2 THEN foo ’
Revised: = 2 THEN foo ’

Original: , we convert it into a language explanation as ‘If number of hands equal to 2, then foo’.
Revised: , we convert it into a language explanation as ‘If number of hands equal to 2, then foo’.

Original: In the presence of quantifiers, we add ‘it is [INSERT QUANTIFIER] ’ before the label.
Revised: In the presence of quantifiers, we add ‘it is [INSERT QUANTIFIER] ’ before the label.

Original: For example if the rule was associated with a quantifier ‘usually’, the language explanation would be ‘If number of hands equal to 2, then it is usually foo’.
Revised: For example if the rule was associated with a quantifier ‘usually’, the language explanation would be ‘If number of hands equal to 2, then it is usually foo’.

Original: B. Label Assignment using Rules 
Revised: A.

Revised: Label Assignment using Rules

Original: In Algorithm 1, we detail the procedure for obtaining label assignments for our synthetic tasks.
Revised: In Algorithm 1, we detail the procedure for obtaining label assignments for our synthetic tasks.

Original: Given that our rules are in an “ IF ... THEN .." format, we split each rule into an antecedent and a consequent based on the position of THEN .
Revised: Given that our rules are in an “ IF ... THEN .." format, we split each rule into an antecedent and a consequent based on the position of THEN .

Original: Note that our voting-based approach to choose the final label for an example helps to tackle (1) negation on a label for multiclass tasks and (2) choose the most suited label in case antecedents from multiple rules are satisfied by an example. 
Revised: Note that our voting-based approach to choose the final label for an example helps to tackle (1) negation on a label for multiclass tasks and (2) choose the most suited label in case antecedents from multiple rules are satisfied by an example. A. Different synthetic task types

Revised: We create our synthetic tasks by varying along the following axes:

Revised: • Number of labels: L = {‘binary’, ‘multiclass’ }

Original: Algorithm
Revised: Algorithm

Original: 1: Given: Task T with rule set R and label set L 2: Votes ← Zeros( | L | ) 3: for rule r ∈ R do 4: r a : Antecedent of r 5: r c : Consequent of r 6: l r ← Label mentioned in r c 7: t ← Truth Value of r a 8: if any quantifier in r then 9: p quant : Prob.
Revised: 1: Given: Task T with rule set R and label set L 2: Votes ← Zeros( | L | ) 3: for rule r ∈ R do 4: r a : Antecedent of r 5: r c : Consequent of r 6: l r ← Label mentioned in r c 7: t ← Truth Value of r a 8: if any quantifier in r then 9: p quant : Prob.

Original: of quantifier from Table 2 10:
Revised: of quantifier from Table 2 10:

Original: Alter l r to any label in L \ l r with probability 11: 1 − p quant 12: end if 13: if t =
Revised: Alter l r to any label in L \ l r with probability 11: 1 − p quant 12: end if 13: if t =

Original: True then 14:
Revised: True then 14:

Original: Votes[ l r ] += 1 15: else 16: for label l ∈
Revised: Votes[ l r ] += 1 15: else 16: for label l ∈

Original: L \ l r do 17:
Revised: L \ l r do 17:

Original: Votes[ l ] += 1 18: end for 19: end if 20: l assigned ← argmax(Votes) 21: end for
Revised: Votes[ l ] += 1 18: end for 19: end if 20: l assigned ← argmax(Votes) 21: end for

Original: B. Different synthetic task types

Original: We create our synthetic tasks by varying along the following axes:

Original: •Number of labels: L = {‘binary’, ‘multiclass’ } • Structure of explanation: C = {‘simple’, ‘conjunction/disjunction’, ‘nested’ } • Presence of quantifier: Q = {‘not present’, ‘present’} • Negation: N = {‘no negation’, ‘negation only in clause’, ‘negation only on label’, ‘negation in clause or on label’} 
Revised: • Structure of explanation: C = {‘simple’, ‘conjunction/disjunction’, ‘nested’ } • Presence of quantifier: Q = {‘not present’, ‘present’} • Negation: N = {‘no negation’, ‘negation only in clause’, ‘negation only on label’, ‘negation in clause or on label’} The

Original: The set of task types is defined as L × C × Q × N , enumerating to 48 different types.
Revised: set of task types is defined as L × C × Q × N , enumerating to 48 different types.

Original: C Real-World Tasks from UCI, Kaggle and Wikipedia
Revised: B Real-World Tasks from UCI, Kaggle and Wikipedia

Original: For our benchmark, we made use of 18 datasets in UCI, 7 datasets in Kaggle, and 9 tables in
Revised: For our benchmark, we made use of 18 datasets in UCI, 7 datasets in Kaggle, and 9 tables in

Original: Wikipedia.
Revised: Wikipedia.

Original: In Table 4, we list the keywords that we use to refer to these tasks along with the URLs to the datasets/tables.
Revised: In Table 4, we list the keywords that we use to refer to these tasks along with the URLs to the datasets/tables.

Original: C. Selecting tasks from UCI and Kaggle
Revised: B. Selecting tasks from UCI and Kaggle

Original: We manually filter the available datasets to avoid ones with (a) many missing attributes and (b) complex attribute names that require extensive domain knowledge making them unsuitable for learning purely from language.
Revised: We manually filter the available datasets to avoid ones with (a) many missing attributes and (b) complex attribute names that require extensive domain knowledge making them unsuitable for learning purely from language.

Original: During pilot studies for collection of explanations for CLUES-Real , we identified that annotators found it difficult to provide explanations for classifications tasks with more than 5 to 6 columns.
Revised: During pilot studies for collection of explanations for CLUES-Real , we identified that annotators found it difficult to provide explanations for classifications tasks with more than 5 to 6 columns.

Original: Appropriately, we reduced the number of columns in most datasets of CLUES-Real to 5 by choosing the top features that had maximum mutual information with the labels in the training dataset.
Revised: Appropriately, we reduced the number of columns in most datasets of CLUES-Real to 5 by choosing the top features that had maximum mutual information with the labels in the training dataset.

Original: The mutual information between the features and the label was computed using the scikit-learn package with a random state of 624.
Revised: The mutual information between the features and the label was computed using the scikit-learn package with a random state of 624.

Original: C. Mining tables from Wikipedia
Revised: B. Mining tables from Wikipedia

Original: Wikipedia is a rich, free source of information readily accessible on the web with a lot of this information stored in a structured format in form of tables
Revised: Wikipedia is a rich, free source of information readily accessible on the web with a lot of this information stored in a structured format in form of tables.

Original: We explore creating additional classification tasks based on tables from Wikipedia, where each row in a table is assigned a category label.
Revised: We explore creating additional classification tasks based on tables from Wikipedia, where each row in a table is assigned a category label.

Original: However, only a small fraction of the tables might be suitable to frame a classification task for our benchmark.
Revised: However, only a small fraction of the tables might be suitable to frame a classification task for our benchmark.

Original: Thus, we need to identify suitable tables by mining a large collection of tables from Wikipedia (we use Wikipedia dump available on April 2021).
Revised: Thus, we need to identify suitable tables by mining a large collection of tables from Wikipedia (we use Wikipedia dump available on April 2021).

Original: We formalize this mining-and-pruning process as a crowdsourcing task (on Amazon Mechanical Turk), where we present each turker with a batch oftables and ask them to pick out suitable tables from that batch.
Revised: We formalize this mining-and-pruning process as a crowdsourcing task (on Amazon Mechanical Turk), where we present each turker with a batch oftables and ask them to pick out suitable tables from that batch.

Original: For a table considered suitable by a turker, we further ask the turker to mention which column of the table should be considered as providing the classification labels.
Revised: For a table considered suitable by a turker, we further ask the turker to mention which column of the table should be considered as providing the classification labels.

Original: D Additional Analysis on Teacher-Student Performance
Revised: C Additional Analysis on Teacher-Student Performance

Original: Figure 3(a) shows the histogram of the Likert ratings (scale 1-4) provided by the students in the explanation verification stage.
Revised: Figure 3(a) shows the histogram of the Likert ratings (scale 1-4) provided by the students in the explanation verification stage.

Original: The average rating for the explanations in CLUES-Real is 2.78, denoting most explanations were useful, even if they did not directly help predict labels in some cases.
Revised: The average rating for the explanations in CLUES-Real is 2.78, denoting most explanations were useful, even if they did not directly help predict labels in some cases.

Original: Figure 3(c) shows average teacher and average student performance on tasks in CLUES-Real .
Revised: Figure 3(c) shows average teacher and average student performance on tasks in CLUES-Real .

Original: Positive correlation ( ρ = 0.48) in this figure indicates that task difficulty (captured by classification accuracy) is well-correlated for a teacher and student on average.
Revised: Positive correlation ( ρ = 0.48) in this figure indicates that task difficulty (captured by classification accuracy) is well-correlated for a teacher and student on average.

Original: For the crowdsourced datasets, we show the number of explanations collected per task in Figure 9(a).
Revised: For the crowdsourced datasets, we show the number of explanations collected per task in Figure 9(a).

Original: The number of explanations is largely around an average value of 11 explanations per task.
Revised: The number of explanations is largely around an average value of 11 explanations per task.

Original: Figure 9(b) shows the relation between explanation quality (quantified by likert scores) and rank of the explanation.
Revised: Figure 9(b) shows the relation between explanation quality (quantified by likert scores) and rank of the explanation.

Original: Rank denotes the order in which a teacher provided that explanation during our crowdsourced explanation collection phase.
Revised: Rank denotes the order in which a teacher provided that explanation during our crowdsourced explanation collection phase.

Original: We find a positive correlation between quality and rank of explanation showing that teachers generally submit most useful explanations (as perceived by them) to teach a task.
Revised: We find a positive correlation between quality and rank of explanation showing that teachers generally submit most useful explanations (as perceived by them) to teach a task.

Original: Finally, we do not observe any correlation between explanation length and ratings as indicated by Figure 9(c).
Revised: Finally, we do not observe any cor- relation between explanation length and ratings as indicated by Figure 9(c).

Original: We also illustrate the differences between teacher and student on our tasks in §3.
Revised: We also illustrate the differences between teacher and student on our tasks in §3.

Original: Here we present two additional plots showing the performance of (1) best teacher vs their students for each task (Figure 7) and (2) worst teacher vs their students for each task (Figure 8).
Revised: Here we present two additional plots showing the performance of (1) best teacher vs their students for each task (Figure 7) and (2) worst teacher vs their students for each task (Figure 8).

Original: We find that even though the best teachers often attain near-perfect accuracies for the tasks, their students perform significantly worse than them in many tasks.
Revised: We find that even though the best teachers often attain near-perfect accuracies for the tasks, their students perform significantly worse than them in many tasks.

Original: The explanations from the worst teachers did not help students in getting significantly better than random performance for majority of the tasks, even though the student did outperform the worst teacher.
Revised: The explanations from the worst teachers did not help students in getting significantly better than random performance for majority of the tasks, even though the student did outperform the worst teacher.

Original: [Figure]
Revised: [Figure]

Original: E Reward Structure for Crowd-sourcing Tasks
Revised: D Reward Structure for Crowd-sourcing Tasks

Original: Our work involves multiple stages of crowdsourcing to collect high-quality explanations for the classification tasks.
Revised: Our work involves multiple stages of crowdsourcing to collect high-quality explanations for the classification tasks.

Original: We pick turkers in the US for explanation collection and verification tasks (US,UK,NZ, and GB for Wikipedia mining Task) with a 98%
Revised: We pick turkers in the US for explanation collection and verification tasks (US,UK,NZ, and GB for Wikipedia mining Task) with a 98%

Original: HIT approval rate and a minimum of 1000 HITs approved.
Revised: HIT approval rate and a minimum of 1000 HITs approved.

Original: In Table 3, we summarize the payment structure provided to the turkers on the AMT platform for each of the stages (described in detail in §2) – (1) Wikipedia mining on tables scraped from[Figure]
Revised: In Table 3, we summarize the payment structure provided to the turkers on the AMT platform for each of the stages (described in detail in §2) – (1) Wikipedia mining on tables scraped from

Original: Wikipedia, (2) Explanation collection for tables obtained from UCI, Kaggle and Wikipedia, and (3) Explanation validation for collected explanations.
Revised: Wikipedia, (2) Explanation collection for tables obtained from UCI, Kaggle and Wikipedia, and (3) Explanation validation for collected explanations.

Original: For all the three crowdsourcing tasks, the  turkers were compensated fairly and the payment per task is equivalent to an hourly compensation that is greater than minimum wage (based on the median time taken by turkers).
Revised: For all the three crowdsourcing tasks, the [Figure] turkers were compensated fairly and the payment per task is equivalent to an hourly compensation that is greater than minimum wage (based on the median time taken by turkers)

Original: [Table]
Revised: [Table]

Original: [Figure] [Table] [Figure]
Revised: WikipediaMining

Original: [Table]
Revised: [Table]

Original: ExplanationCollection
Revised: ExplanationCollection

Original: [Table]
Revised: [Table]

Original: ExplanationValidation$1.
Revised: ExplanationValidation

Revised: [Table]

Original: F Training details
Revised: E Training details

Original: In this section we proved details about implementation of various models, hyperparameter details, and details about hardware and software used along with an estimate of time taken to train the models.
Revised: In this section we proved details about implementation of various models, hyperparameter details, and details about hardware and software used along with an estimate of time taken to train the models.

Original: Code and dataset for our paper will be made public upon first publication.
Revised: Code and dataset for our paper will be made public upon first publication.

Original: F. Details of seen and novel tasks for CLUES-Real and CLUES-Synthetic
Revised: E. Details of seen and novel tasks for CLUES-Real and CLUES-Synthetic

Original: For CLUES-Real , we chose the tasks from
Revised: For CLUES-Real , we chose the tasks from

Original: Wikipedia that have very examples to be part of novel task set.
Revised: Wikipedia that have very examples to be part of novel task set.

Original: Among the tasks from Kaggle and
Revised: Among the tasks from Kaggle and

Original: [Figure]
Revised: UCI, we kept tasks with higher number of samples as part of seen tasks (training tasks).

Original: e
Revised: Seen tasks (20) for CLUES-Real are: • website-phishing

Original: [Figure]
Revised: [Figure]

Original: c
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: UCI, we kept tasks with higher number of samples as part of seen tasks (training tasks). Seen tasks (20) for CLUES-Real are:
Revised: [Figure]

Original: • website-phishing • internet-firewall • mushroom • dry-bean • wine • caesarian-section • occupancy-detection • vertebral-column •
Revised: [Figure] A v e [Figure] age

Revised: L i k e r t

Revised: [Figure]

Revised: c o r e = 0.

Revised: Likert Score vs Explanation Length

Revised: [Figure]

Revised: • internet-firewall • mushroom • dry-bean • wine • caesarian-section • occupancy-detection • vertebral-column •

Original: student-performance • shill-bidding • mammographic-mass • teaching-assistant-evaluation • somerville-happiness • stroke-prediction • job-change • campus-placement • engineering-placement • water-potability • color-luminance • proteinogenic-acid
Revised: student-performance • shill-bidding • mammographic-mass • teaching-assistant-evaluation • somerville-happiness • stroke-prediction • job-change • campus-placement • engineering-placement • water-potability • color-luminance • proteinogenic-acid

Original: Novel tasks (16) for CLUES-Real are:
Revised: Novel tasks (16) for CLUES-Real are:

Original: • banknote-authentication • tic-tac-toe-endgame • car-evaluation •
Revised: • banknote-authentication • tic-tac-toe-endgame • car-evaluation •

Original: contraceptive-method-choice • indian-liver-patient • travel-insurance • entrepreneur-competency • award-nomination-result • coin-face-value • coin-metal • driving-championship-points • election-outcome • hotel-rating • manifold-orientability • soccer-club-region • soccer-league-type
Revised: contraceptive-method-choice • indian-liver-patient • travel-insurance • entrepreneur-competency • award-nomination-result • coin-face-value • coin-metal • driving-championship-points • election-outcome • hotel-rating • manifold-orientability • soccer-club-region • soccer-league-type

Original: We train on 70% of the labeled examples of the seen tasks and perform zero-shot generalization test over the 20% examples of each task in
Revised: We train on 70% of the labeled examples of the seen tasks and perform zero-shot generalization test over the 20% examples of each task in

Original: CLUES-Real .
Revised: CLUES-Real .

Original: For the extremely small Wikipedia tasks (for which we do not crowdsource explanations), we use the entire set of examples for zeroshot testing.
Revised: For the extremely small Wikipedia tasks (for which we do not crowdsource explanations), we use the entire set of examples for zeroshot testing.

Original: For CLUES-Synthetic , we have 96 tasks as seen (training) tasks and 48 as novel tasks.
Revised: For CLUES-Synthetic , we have 96 tasks as seen (training) tasks and 48 as novel tasks.

Original: Task in CLUES-Synthetic that belong to the following schemas are part of the seen tasks:
Revised: Task in CLUES-Synthetic that belong to the following schemas are part of the seen tasks:

Original: • Species of Animal • Species of Bird • Rainfall prediction
Revised: • Species of Animal • Species of Bird • Rainfall prediction

Original: Tasks belonging to ‘Bond relevance classification’ and ‘League Rank Classification’ were part of novel tasks for CLUES-Synthetic .
Revised: Tasks belonging to ‘Bond relevance classification’ and ‘League Rank Classification’ were part of novel tasks for CLUES-Synthetic .

Original: We train on 700 labeled examples of each seen task and perform zero-shot generalization test over 200 examples of each novel task in CLUES-Synthetic .
Revised: We train on 700 labeled examples of each seen task and perform zero-shot generalization test over 200 examples of each novel task in CLUES-Synthetic .

Original: F. Model parameters
Revised: E. Model parameters

Original: • RoBERTa w/o Exp.
Revised: • RoBERTa w/o Exp.

Original: : The number of parameters is same as the pretrained RoBERTa-base model available on HuggingFace library.
Revised: : The number of parameters is same as the pretrained RoBERTa-base model available on HuggingFace library.

Original: • RoBERTa w/ Exp. :
Revised: • RoBERTa w/ Exp.:

Original: The number of parameters is same as the pretrained RoBERTa-base model available on HuggingFace library.
Revised: The number of parameters is same as the pretrained RoBERTa-base model available on HuggingFace library.

Original: • ExEnt : We consider a pre-trained RoBERTa model fine-tuned on MNLI (Williams et al., 2017) corpus as our base entailment model.
Revised: • ExEnt : We consider a pre-trained RoBERTa model fine-tuned on MNLI (Williams et al., 2017) corpus as our base entailment model.

Original: The number of parameters in ExEnt is same as the pre-trained RoBERTa mdoel finetuned on
Revised: The number of parameters in ExEnt is same as the pre-trained RoBERTa mdoel finetuned on

Original: MNLI corpus.
Revised: MNLI corpus.

Original: Further, in order to perform the assignment of logits using an explanation, we maintain meta-information for each explanation to (1) determine if the explanation mentions to ‘assign’ a label or ‘not assign’ a label, and (2) track l exp (label mentioned in explanation).
Revised: Further, in order to perform the assignment of logits using an explanation, we maintain meta-information for each explanation to (1) determine if the explanation mentions to ‘assign’ a label or ‘not assign’ a label, and (2) track l exp (label mentioned in explanation).

Original: For
Revised: For

Original: CLUES-Synthetic , we parse the templated explanations to obtain the meta-information, while for the explanations in CLUES-Real , the authors manually annotate this meta-information.
Revised: CLUES-Synthetic , we parse the templated explanations to obtain the meta-information, while for the explanations in CLUES-Real , the authors manually annotate this meta-information.

Original: F. Hyper-parameter settings
Revised: E. Hyper-parameter settings

Original: For all the transformer based models we use the implementation of HuggingFace library (Wolf et al., 2020).
Revised: For all the transformer based models we use the implementation of HuggingFace library (Wolf et al., 2020).

Original: All the model based hyper-parameters are thus kept default to the settings in the HuggingFace library.
Revised: All the model based hyper-parameters are thus kept default to the settings in the HuggingFace library.

Original: We use the publicly available checkpoints to initialise the pre-trained models.
Revised: We use the publicly available checkpoints to initialise the pre-trained models.

Original: For RoBERTa based baselines we use ‘roberta-base’ checkpoint available on HuggingFace.
Revised: For RoBERTa based baselines we use ‘roberta-base’ checkpoint available on HuggingFace.

Original: For our intermediate entailment model in ExEnt , we finetune a pretrained checkpoint of RoBERTa trained on MNLI corpus (‘textattack/roberta-base-MNLI’)
Revised: For our intermediate entailment model in ExEnt , we finetune a pretrained checkpoint of RoBERTa trained on MNLI corpus (‘textattack/roberta-base-MNLI’)

Original: When training on CLUES-Synthetic , we use a maximum of 64 tokens for our baseline RoBERTa w/o Exp. and ExEnt .
Revised: When training on CLUES-Synthetic , we use a maximum of 64 tokens for our baseline RoBERTa w/o Exp. and ExEnt .

Original: For the RoBERTa w/ Exp.
Revised: For the RoBERTa w/ Exp.

Original: model we increase this limit to 128 tokens as it takes concatenation of all explanations for a task.
Revised: model we increase this limit to 128 tokens as it takes concatenation of all explanations for a task.

Original: When training on CLUES-Real , we use 256 tokens as limit for RoBERTa w/ Exp.
Revised: When training on CLUES-Real , we use 256 tokens as limit for RoBERTa w/ Exp.

Original: using explanations as the real-world tasks have roughly two times more explanations on average than synthetic tasks.
Revised: using explanations as the real-world tasks have roughly two times more explanations on average than synthetic tasks.

Original: We used the AdamW (Loshchilov and Hutter, 2019) optimizer commonly used to fine-tune pretrained Masked Language Models (MLM) models.
Revised: We used the AdamW (Loshchilov and Hutter, 2019) optimizer commonly used to fine-tune pretrained Masked Language Models (MLM) models.

Original: For fine-tuning the pre-trained models on our benchmark tasks, we experimented with learning rates { 1 e − 5 , 2 e − 5 } and chose 1 e − 5 based on performance on the performance on the validation set of seen tasks.
Revised: For fine-tuning the pre-trained models on our benchmark tasks, we experimented with learning rates { 1 e − 5 , 2 e − 5 } and chose 1 e − 5 based on performance on the performance on the validation set of seen tasks.

Original: Batch sizes was kept as 2 with gradient accumulation factor of 8.
Revised: Batch sizes was kept as 2 with gradient accumulation factor of 8.

Original: The random seed for all experiments was 42.
Revised: The random seed for all experiments was 42.

Original: We train all the models for 20 epochs.
Revised: We train all the models for 20 epochs.

Original: Each epoch comprises of 100 batches, and in each batch the models look at one of the tasks (in a sequential order) in the seen split.
Revised: Each epoch comprises of 100 batches, and in each batch the models look at one of the tasks (in a sequential order) in the seen split.

Original: F. ExEnt Implementation Details
Revised: E. ExEnt Implementation Details

Original: In experiments, we consider a pre-trained
Revised: In experiments, we consider a pre-trained

Original: RoBERTa model fine-tuned on MNLI (Williams et al., 2017) corpus as our base entailment model.
Revised: RoBERTa model fine-tuned on MNLI (Williams et al., 2017) corpus as our base entailment model.

Original: Further, in order to perform the assignment of logits using an explanation, we maintain metainformation for each explanation to (1) determine if the explanation mentions to ‘assign’ a label or ‘not assign’ a label, and (2) track l exp (label mentioned in explanation).
Revised: Further, in order to perform the assignment of logits using an explanation, we maintain metainformation for each explanation to (1) determine if the explanation mentions to ‘assign’ a label or ‘not assign’ a label, and (2) track l exp (label mentioned in explanation).

Original: For CLUES-Synthetic , we parse the templated explanations to obtain the meta-information, while for the explanations in
Revised: For CLUES-Synthetic , we parse the templated explanations to obtain the meta-information, while for the explanations in

Original: CLUES-Real , the authors manually annotate this meta-information.
Revised: CLUES-Real , the authors manually annotate this meta-information.

Original: F. Hardware and software specifications
Revised: E. Hardware and software specifications

Original: All the models are coded using Pytorch 1.4.0(Paszke et al., 2019) and related libraries like numpy (Harris et al., 2020), scipy (Jones et al., 2001–) etc.
Revised: All the models are coded using Pytorch 1.4.0(Paszke et al., 2019) and related libraries like numpy (Harris et al., 2020), scipy (Jones et al., 2001–) etc.

Original: We run all experiments on one of the following two systems - (1) GeForce RTX
Revised: We run all experiments on one of the following two systems - (1) GeForce RTX

Original: GPU of size 12 GB, 256 GB RAM and 40 CPU cores (2) Tesla V100-SXM2 GPU of size 16GB, 250 GB RAM and 40 CPU cores.
Revised: GPU of size 12 GB, 256 GB RAM and 40 CPU cores (2) Tesla V100-SXM2 GPU of size 16GB, 250 GB RAM and 40 CPU cores.

Original: F. Training times
Revised: E. Training times

Original: • Training on CLUES-Real : The baseline RoBERTa w/ o Exp model typically takes 3 seconds on average for training on 1 batch of examples.
Revised: • Training on CLUES-Real : The baseline RoBERTa w/o Exp model typically takes 3 seconds on average for training on 1 batch of examples.

Original: In 1 batch, the model goes throughexamples from the tasks in seen split.
Revised: In 1 batch, the model goes throughexamples from the tasks in seen split.

Original: RoBERTa w/ Exp. takes around 5 seconds to train onbatch.
Revised: RoBERTa w/ Exp. takes around 5 seconds to train onbatch.

Original: ExEnt takes longer time than baselines owing to the multiple forward passes.
Revised: ExEnt takes longer time than baselines owing to the multiple forward passes.

Original: For training on 1 batch of CLUES-Real , ExEnt took seconds on average.
Revised: For training on 1 batch of CLUES-Real , ExEnt tookseconds on average.

Original: • Training on CLUES-Synthetic : All the models take comparatively much lesser time for training on our synthetic tasks owing to lesser number of explanations on average for a task.
Revised: • Training on CLUES-Synthetic : All the models take comparatively much lesser time for training on our synthetic tasks owing to lesser number of explanations on average for a task.

Original: For training on 1 batch, all models took1 seconds or less to train on 1 batch of examples from
Revised: For training on 1 batch, all models took seconds or less to train on 1 batch of examples from

Original: CLUES-Synthetic
Revised: CLUES-Synthetic

Original: G Key Challenges
Revised: F Key Challenges

Original: We identify the open challenges in learning from explanations by ablating the linguistic components 
Revised: We identify the open challenges in learning from explanations by ablating the linguistic components and structure of explanations.

Revised: For a robust analysis, we generate more tasks for each task type in

Revised: CLUES-Synthetic , making 100 tasks for each of the 48 different task-types in CLUES-Synthetic

Original: [Figure]
Revised: [Figure]

Original: R e l a t i v e
Revised: R e l a t i v e

Original: P e
Revised: P e

Original: r
Revised: r

Original: f
Revised: f

Original: o r m a n c e G a i n (
Revised: o r m a n c e G a i n (

Original: [Figure]
Revised: [Figure]

Original: and structure of explanations.

Original: For a robust analysis, we generate more tasks for each task type in

Original: CLUES-Synthetic , making 100 tasks for each of the 48 different task-types in CLUES-Synthetic (axes of variation include 4 negation types, 3 conjunction/disjunction types, 2 quantifier types, and number of labels; details in Appendix B.5).

Original: We evaluate the generalization performance of
Revised: We evaluate the generalization performance of

Original: ExEnt to novel tasks on each of the different types separately by training separate models for each task type.
Revised: ExEnt to novel tasks on each of the different types separately by training separate models for each task type.

Original: Figure 10 shows the relative gain in generalization performance of models learned using explanations compared to the performance of baseline RoBERTa w/o Exp.
Revised: Figure 10 shows the relative gain in generalization performance of models learned using explanations compared to the performance of baseline RoBERTa w/o Exp.

Original: Our results indicate that learning from explanations containing quantifiers is highly challenging.
Revised: Our results indicate that learning from explanations containing quantifiers is highly challenging.

Original: In the presence of quantifiers, models guided by explanations perform on par with the baseline RoBERTa w/o Exp model.
Revised: In the presence of quantifiers, models guided by explanations perform on par with the baseline RoBERTa w/o Exp model.

Original: Negations also pose a challenge, as indicated by the decline in relative gains of models guided by explanation compared to the RoBERTa w/o Exp model.
Revised: Negations also pose a challenge, as indicated by the decline in relative gains of models guided by explanation compared to the RoBERTa w/o Exp model.

Original: Structurally complex explanations (containing conjunctions/disjunctions of clauses) are also hard to learn from compared to simple conditional statements.
Revised: Structurally complex explanations (containing conjunctions/disjunctions of clauses) are also hard to learn from compared to simple conditional statements.

Original: These challenges provide a fertile ground for exciting future research.
Revised: These challenges provide a fertile ground for exciting future research.

Original: H Annotation interfaces
Revised: G Annotation interfaces

Original: We present the different annotation templates and interfaces used for our explanation collection and verification stages in Figures 11,12,13,14 and Figure 15 respectively.
Revised: We present the different annotation templates and interfaces used for our explanation collection and verification stages in Figures 11,12,13,14 and Figure 15 respectively.

Original: D ATASET
Revised: D ATASET

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: authentication
Revised: authentication

Original: [Figure]
Revised: [Figure]

Original: choice
Revised: choice

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: caesarian-
Revised: caesarian-

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] 
Revised: [Table] [Table] [Table] [Table]

Original: [Table] [Table] 
Revised: [Table] [Table] [Table]

Original: [Table] [Table] 
Revised: [Table] [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table][Table] [Table] [Table]
Revised: [Table]

Original: [Table] [Table][Table]
Revised: [Table] [Table]

Original: [Table] [Table][Table]
Revised: [Table] [Table]