CLUES : A Benchmark for Learning Classifiers using Natural Language Explanations (2022)

Abstract: Supervised learning has traditionally focused on inductive learning by observing labeled examples of a task. In contrast, a hallmark of human intelligence is the ability to learn new concepts purely from language. Here, we explore training zero-shot classifiers for structured data purely from language. For this, we introduce CLUES, a benchmark for Classifier Learning Using natural language ExplanationS, consisting of a range of classification tasks over structured data along with natural language supervision in the form of explanations. CLUES consists of 36 real-world and 144 synthetic classification tasks. It contains crowdsourced explanations describing real-world tasks from multiple teachers and programmatically generated explanations for the synthetic tasks. To model the influence of explanations in classifying an example, we develop ExEnt, an entailment-based model that learns classifiers using explanations. ExEnt generalizes up to 18% better (relative) on novel tasks than a baseline that does not use explanations. We delineate key challenges for automated learning from explanations, addressing which can lead to progress on CLUES in the future. Code and datasets are available at: https://clues-benchmark.github.io .

## introduction humans have a remarkable ability to learn concepts through language @xcite . for example, we can learn about poisonous mushrooms through an explanation like 'a mushroom is poisonous if it has pungent odor'. such figure 1 : we explore learning classification tasks over structured data from natural language supervision in form of explanations. the explanations provide declarative supervision about the task, and are not examplespecific. this is an example from the uci mushroom dataset, one of the 36 real-world datasets for which we collect multiple sets of explanations in clues. an approach profoundly contrasts with the predominant paradigm of machine learning, where algorithms extract patterns by looking at scores of labeled examples of poisonous and edible mushrooms. however, it is unnatural to presume the availability of labeled examples for the heavy tail of naturally occurring concepts in the world. this work studies how models trained to learn from natural language explanations can generalize to novel tasks without access to labeled examples. while prior works in this area @xcite @xcite @xcite @xcite have explored explanations as a source of supervision, they evaluate models on a small number of tasks (2-3 relation extraction tasks in @xcite @xcite , 7 email categorization tasks @xcite ). owing to the paucity of large-scale benchmarks for learning from explanations over diverse tasks, we develop clues, a benchmark of classification tasks paired with natural language explanations. over the last few decades, researchers and engineers alike have put immense effort into constructing structured and semi-structured knowledge bases (e.g., structured tables on wikipedia, e-commerce sites, etc.). developing models that can reason over structured data is imperative to improve the accessibility of machine learning models, enabling even non-experts to interact with such data. hence, in this work, we specifically formulate our classification tasks over structured data. our benchmark is divided into clues-real and clues-synthetic consisting of tasks from real-world (uci, kaggle, and wikipedia) and synthetic domains respectively. explanations for clues-real are crowdsourced to mimic the diversity and difficulty of human learning and pedagogy. for clues-synthetic, we generate the explanations programmatically to explicitly test models' reasoning ability under a range of structural and linguistic modifications of explanations. we train models with a mix of explanations and labeled examples, in a multi-task setup, over a set of seen classification tasks to induce generalization to novel tasks, where we do not have any labeled examples. @xcite refer to this problem setup as "cross-task generalization". some recent methods on cross-task generalization from language use instructions/prompts @xcite @xcite describing information about 'what is the task?' to query large language models. in contrast, language explanations in clues provide the logic for performing the classification task, or intuitively 'how to solve the task?'. for the running example of mushroom classification, an instruction/prompt might be 'can you classify a mushroom with pungent odor as poisonous or edible?'. on the other hand, an example of an explanation in clues is 'a mushroom is poisonous if it has pungent odor'. we find that simply concatenating explanations to the input does not help pre-trained models, like roberta @xcite , generalize to new tasks. thus, we develop exent, an entailmentbased model for learning classifiers guided by explanations, which explicitly models the influence of individual explanations in deciding the label of an example. exent shows a relative improvement of up to 18% over other baselines on unseen tasks. to identify the challenges of learning from explanations, we perform extensive analysis over synthetic tasks. our analysis explores how the structure of an explanation (simple clauses vs. nested clauses) and the presence of different linguistic components in explanation (conjunctions, disjunctions, and quantifiers) affect the generalization ability of models. the rest of the paper is structured as follows: we describe our crowdsourced-benchmark creation pipeline in §3. in §4, we analyze our collected data. in §5, we describe our models, experiments, and results. we conclude with a brief discussion on the contributions and our findings, followed by a statement of ethics and broader impact. our contributions are: • we introduce clues, a benchmark for learning classifiers over structured data from language. exent shows a relative improvement of up to 18% over other baselines on generalization to novel tasks. ## related work learning concepts from auxiliary information: prior work has explored techniques to incorporate 'side-information' to guide models during training @xcite . more recently, researchers have explored using language in limited data settings for learning tasks such as text classification @xcite @xcite ) and question answering @xcite . however, we diverge from these works by exploring the generalization ability of classifiers learned by using language over novel tasks as opposed to gauging performance only on seen tasks. ## creating clues in this section, we describe our benchmark creation process in detail. in clues, we frame classification tasks over structured data represented in tabular format. based on the source of tables used to construct the classification tasks, we consider two splits of our benchmark, clues-real (real-world datasets) and clues-synthetic (synthetic datasets). ## dataset analysis in this section, we describe the tasks and the collected explanations in clues. task statistics: as seen in category example real syn generic being over 50 increases the risk of a stroke. 48 % 50 % quantifier ... usually means you won't have heart disease. 52 % 50 % conditional if color code ... , then ... 15 % 100 % negations ... is not low. 16 % 50% to 2, then it is usually foo", will be categorized both as having both conditional and quantifiers.) we found that around 52% of the explanations for the real-world tasks had quantifiers (such as 'some', 'majority', 'most', etc.) in them. a full list of quantifiers present in the data is given in appendix a. ## experiment setup and models in this section, we describe our training and evaluation setup, our models, and experimental findings. ## key challenges to identify key challenges in learning from explanations, we perform experiments ablating the linguistic components and structure of explanations. for a robust analysis, we generate more tasks for each task type in clues-synthetic, making 100 tasks for each of the 48 different task-types in clues-synthetic (axes of variation include 4 negation types, 3 conjunction/disjunction types, 2 ## conclusion we have introduced clues, a benchmark with diverse classification tasks over structured data along with natural language explanations to learn them. clues is agnostic in the domain of tasks allowing the research community to contribute more tasks in the future. we also present exent, an entailmentbased model to learn classifiers guided by explanations. our results are promising and indicate that explicitly modeling the role of each explanation through entailment can enable learning classifiers for new tasks from explanations alone. future work can explore the open challenges in learning from explanations, such as modeling the influence of quantifiers and negations present in an explanation. our empirical analyses here aggregates explana11 accuracies have been averaged over the multi-class and binary datasets since the trends remain the same across both.