When Can Models Learn From Explanations? A Formal Framework for Understanding the Roles of Explanation Data (2022)

Abstract: Many methods now exist for conditioning models on task instructions and user-provided explanations for individual data points. These methods show great promise for improving task performance of language models beyond what can be achieved by learning from individual (x,y) pairs. In this paper, we (1) provide a formal framework for characterizing approaches to learning from explanation data, and (2) we propose a synthetic task for studying how models learn from explanation data. In the first direction, we give graphical models for the available modeling approaches, in which explanation data can be used as model inputs, as targets, or as a prior. In the second direction, we introduce a carefully designed synthetic task with several properties making it useful for studying a model’s ability to learn from explanation data. Each data point in this binary classification task is accompanied by a string that is essentially an answer to the why question: “why does data point x have label y?” We aim to encourage research into this area by identifying key considerations for the modeling problem and providing an empirical testbed for theories of how models can best learn from explanation data.

## introduction a long line of past work has sought to use freetext explanations, rationales, and other similar data to improve machine learning models. proposed methods use explanations to constrain or regularize the learned model @xcite @xcite @xcite , to automatically label data for data augmentation @xcite @xcite , as additional supervision @xcite : hypothetical data and explanations. here, x is an input that one might expect a model to produce the correct output for after fitting to (x, y) pairs. for some models, x may be sufficient, while others may benefit from additional information provided by e. @xcite or intermediate structured variables @xcite @xcite , and simply as model inputs @xcite @xcite . however, there are many tasks in nlp where improvements in performance prove elusive even when using thousands of explanations as additional data @xcite . a few observations could explain this situation: (1) the modeling space has not been fully explored for these tasks, but improvements are possible; (2) pretrained language models already store the knowledge that the explanations would have provided, so they do not need them; (3) the language models do not need any information that is not already learnable from the task's input-output pairs. we do not yet know which explanation is best, and therefore it would be helpful to more deeply understand the motivations behind existing modeling approaches. in this paper, we (1) present a formal framework for characterizing approaches to learning from explanation data, and (2) we propose a synthetic task for studying how models learn from natural language data. specifically, we first present graphical models for various approaches where explanation data is used either as model inputs, targets, or priors, and we characterize existing methods according to these graphical models. then, based on past results, we suggest which models might be most appropriate for explanation data. next, we present a synthetic task which shares important properties with nlp tasks involving explanation data. constructing this task helps us carefully specify the manner in which we expect explanations to be useful to models. we provide simple experimental verification that the task is solvable by existing transformer models when using explanations as additional data but very difficult to solve without them. our aim is to outline promising approaches in the area and contribute a concrete test bed to assist others in developing new models for learning from natural language explanations. ## formalizing the roles of explanations in what follows, we discuss our framework for modeling with explanations and relevant work (sec. 2.1), as well as promising approaches for learning from explanations (sec. 2.2). what is an explanation? we use the term "explanation" to refer to the data one might collect if asking a person to answer the question, "why does data point x have label y?" this is a formulation of the explanation as an answer to a why-question of the kind discussed in @xcite . rather than try to give a formal definition of the kind of data generated from this question, we proceed with some illustrative examples, shown in fig. 1 . ## synthetic task following recent work using synthetic data to investigate sequence modeling questions @xcite , we design a synthetic dataset so that we can carefully control several important data properties. in fig. 3 , we show an example data point and description of how it gets its label. the premise of our task is to classify sequences by counting different integers in them. core idea behind data. we wish to design a task where, for a data point (x, y), an explanation analogous components to real data index an easily computable feature connecting the input to its explanation indicator a feature indicating what information from the explanation is relevant for the input's label an explanation that says why the input received its label, when understood properly synthetic task description: the sequence has label because there are more s than s. the index maps to , and indicator says to count rather than . if there were more s than s, the label would be . there is a one-to-one map between index values and tuples. count whether there are more of integer a than integer b e communicates information about why input x receives label y. the premise of the task is that a binary label for a sequence of integers x is determined by whether there are more of an integer a in the sequence than there are of an integer b. we refer to integers (a, b) that need to be counted as the label reason. this label reason forms the basis of the explanation for each data point, and it is always exactly specified by the first two integers in x, which we term the index and indicator. for every data point x, there is an explanation @xmath0, m, n, r, d) where the label reason is given by either (m, n) or (r, d). whether the label reason is the (m, n) integer pair or the (r, d) pair is dictated by the indicator. as represented in fig. 3 , (a, b) = (m, n) if the indicator is 1 and (a, b) = (r, d) if the indicator is 2. we call the data e an explanation because it is a direct encoding of a natural language explanation for the data (x, y). for the data point in fig. 3 , this natural language explanation is "input x receives label 1 because it contains more 80's than 40's, and we do not need to count 17's or 27's for this sequence." proposed dataset. we describe the proposed dataset using some default data parameters for preliminary experiments, but any specific numbers appearing below are easily adjusted. see supplement d for the full generative process. 1. train set: 5000 sequences of 20 integers (including index and indicator), each accompanied by an explanation. there are 500 unique values of index in the dataset drawn from unif (1, 10000), so there are 10 points for each index, whose values of m, n, r, and d are drawn from unif (1, 100) while requiring that @xmath1. the corresponding 10 values of indicator are split between 1 and 2. half of the points have label @xmath2, i.e. either #@xmath3, depending on which feature is causal. in each x i , after m, n, r, and d have been randomly placed into the sequence, unfilled slots are filled with samples from unif (1, 100). 2. dev set: 10,000 points, none appearing in train, with the same 500 index values, and twice the number of points per index as train. 3. test set: 50,000 points of similar construction to the dev set, but with five times the points per index as train. analogous properties to human-curated data. we claim that aspects of our synthetic task are analogous to properties that natural language data might take on, which we represent in fig. 3 . first, e is an explanation in the sense that, when understood properly, it is a plausible answer to the question: "why does point x have label y?" the explanation describes the feature that causes the label, i.e. the integers that should be counted. we suggest that the index in a sequence is analogous to the topic of some text or the things it refers to: it is an easily computable feature that connects the input to the appropriate explanation. meanwhile, the indicator is a feature that tells how information from an explanation is relevant to deciding the label. similarly, an explanation might only be understood in the context of the input it explains. ## initial experiments we include experiments below that (1) show explanation data is helpful for solving our task and (2) demonstrate why the task is hard without explanation data. we make use of a retrieval-based model similar to @xcite , which learns to retrieve explanations from the training dataset to help with prediction at test time (details in appendix b and c). this model is composed of a robertabase classifier @xcite and a sentencer-oberta model used for retrieval (reimers and gurevych, 2019). the baseline in our experiments is the roberta classifer on its own. ## discussion & conclusion we present a synthetic dataset with key similarities to natural language explanation data, and we show that our explanations are highly useful for model learning. however, we emphasize that if a model already "knew" the information in some explanations, it might not need them. this may plausibly occur with sufficiently large pretrained models that store a great deal of factual knowledge @xcite . similarly, the necessary information might be learnable from (x, y ) data alone. future work on modeling approaches we outline in this paper (fig. 2 ) will benefit from testing their methods on controlled synthetic tasks as a test of their ability to learn from explanation data. then, further analysis will be helpful for understanding how explanations contain novel information that is not learned elsewhere in pretraining or finetuning.