Rationales for Sequential Predictions (2021)

Abstract: Sequence models are a critical component of modern NLP systems, but their predictions are difficult to explain. We consider model explanations though rationales, subsets of context that can explain individual model predictions. We find sequential rationales by solving a combinatorial optimization: the best rationale is the smallest subset of input tokens that would predict the same output as the full sequence. Enumerating all subsets is intractable, so we propose an efficient greedy algorithm to approximate this objective. The algorithm, which is called greedy rationalization, applies to any model. For this approach to be effective, the model should form compatible conditional distributions when making predictions on incomplete subsets of the context. This condition can be enforced with a short fine-tuning step. We study greedy rationalization on language modeling and machine translation. Compared to existing baselines, greedy rationalization is best at optimizing the sequential objective and provides the most faithful rationales. On a new dataset of annotated sequential rationales, greedy rationales are most similar to human rationales.

## introduction sequence models are a critical component of generation tasks ranging from language modeling to machine translation to summarization. these tasks are dominated by complex neural networks. while these models produce accurate predictions, their decision making processes are hard to explain. interpreting a model's prediction is important in a variety of settings: a researcher needs to understand a model to debug it; a doctor using a diagnostic model requires justifications to validate a decision; a company deploying a language model relies on model explanations to detect biases appropriated from training data. interpretation takes many flavors @xcite . we focus on rationales, i.e. identifying the most the supreme court on tuesday rejected a challenge to the constitutionality of the death penalty the supreme court on tuesday rejected a challenge to the constitutionality of the death penalty figure 1 . rationales for sequential prediction on gpt-2. each row is a predicted word. the dark cells correspond to the context words found by greedy rationalization. to predict "constitutionality", the model only needs "supreme", "court", "challenge", and "the". various rationale methods have been proposed for sequence classification, where each sequence has a single rationale @xcite @xcite . however, these methods cannot scale to sequence models, where each token in a sequence requires a different rationale. this work frames the problem of finding se-quence rationales as a combinatorial optimization: given a model, the best rationale is the smallest subset of input tokens that would predict the same token as the full sequence. finding the global optimum in this setting is intractable, so we propose greedy rationalization, a greedy algorithm that iteratively builds longer rationales. this approach is efficient for many nlp models such as transformers @xcite . moreover, it does not require access to the inner workings of a model, such as gradients. underlying this approach is an assumption that the model forms sensible predictions for incomplete subsets of the input. although we can pass in incomplete subsets to neural models, there is no guarantee that their predictions on these subsets will be compatible with their predictions on full contexts @xcite . we show that compatibility can be learned by conditioning on randomly sampled context subsets while training a model. for large pretrained models like gpt-2 @xcite , fine-tuning is sufficient. in an empirical study, we compare greedy rationalization to various gradient-and attention-based explanation methods on language modeling and machine translation. greedy rationalization best optimizes the objective, and its rationales are most faithful to the inner workings of the model. we additionally create a new dataset of annotated rationales based on the lambada corpus @xcite . we find that greedy rationales are most similar to human annotations, both on our dataset and on a labeled dataset of translation alignments. our code and annotated dataset are available. 1 here, f θ is the specific model parameterizing p θ , such as a transformer @xcite , and is trained to take inputs @xmath0. going forward, we drop the dependence on θ in the notation. word-level explanations are a natural way to interpret a sequence model: which words were instrumental for predicting a particular word? would the same word have been predicted if some of the words had been missing? explanations may be straightforward for simpler models; for example, a bigram markov model uses only the previously generated word to form predictions. however, the most effective sequence models have been based on neural networks, whose predictions are challenging to interpret @xcite . motivated by this goal, we consider a sequence y 1:t generated by a sequence model p. at each position t, the model takes the inputs in the context @xmath1. we are interested in forming rationales: subsets of the contexts that can explain the model's prediction of y t . 2 what are the properties of a good rationale? any of the contextual words @xmath2. however, if a model makes the same prediction with only a subset of the context, that subset contains explanatory power on its own. a rationale is sufficient if the model would produce the same y t having seen only the rationale @xcite . while rationales consisting of the full context would always be sufficient, they would be ineffective for explaining longer sequences. intuitively, the smaller the rationale, the easier it is to interpret, so we also prioritize brevity. we combine these desiderata and frame finding rationales as a combinatorial optimization: the best rationale of a word y t is the smallest subset of inputs that would lead to the same prediction. each candidate rationale s is an index set, and y s denotes the subset of tokens indexed by s. denote by @xmath3. an optimal rationale is given by the constraint guarantees sufficiency, and the objective targets brevity. although the objective may have multiple solutions, we only require one. optimizing equation 3 is hindered by a pair of computational challenges. the first challenge is that solving this combinatorial objective is intractable; framed as a decision problem, it is nphard. we discuss this challenge in section 3. the second challenge is that evaluating distributions conditioned on incomplete context subsets p(y t |y s ) involves an intractable marginalization over missing tokens. for now we assume that f (y t |y s ) ≈ p(y t |y s ); we discuss how to enforce this condition in section 4. ## greedy rationalization we propose a simple greedy algorithm, greedy rationalization, to approximate the solution to equation 3. the algorithm starts with an empty rationale. at each step, it considers adding each possible token, and it selects the one that most increases the probability of y t . this process is repeated until the rationale is sufficient for predicting y t . figure 2 provides an overview. here is the algorithm. begin with a rationale s (0) = ∅. denoting by [t -1] = {1, . . . , t -1}, the first rationale set is at each step, we iteratively add a single word to the rationale, choosing the one that maximizes the probability of the word y t : we continue iterating equation 5 until arg max y t p(y t |y s (n) ) = y t . the procedure will always converge, since in the worst case, s (t-1) contains the full context. the greedy approach is motivated by approximations to the set cover problem @xcite . in our setting, each set is a single context token, and a rationale "covers" a sequence if it results in predicting the same token. this procedure is simple to implement, and it is black-box: it does not require access to the inner workings of a model, like gradients or attention. while greedy rationalization can be applied to any model, greedy rationalization is particularly effective for set-based models such as transformers. if we assume the rationale size @xmath4, greedy rationalization requires no extra asymptotic complexity beyond the cost of a single evaluation. for transformers, the complexity of each evaluation f (y t |@xmath5. ## model compatibility greedy rationalization requires computing conditional distributions p(y t |y s ) for arbitrary subsets s. using an autoregressive model, this calculation requires marginalizing over unseen positions. for example, rationalizing a sequence y 1:3 requires evaluating the candidate rationale p(y 3 |y 1 ), which marginalizes over the model's predictions: given the capacity of modern neural networks, it is tempting to pass in incomplete subsets y s to f and evaluate this instead as f (y t |y s ) ≈ p(y t |y s ). however, since f is trained only on complete feature subsets @xmath6, incomplete feature subsets y s are out-of-distribution @xcite . evaluating f (y 3 |y 1 ) may be far from the true conditional p(y 3 |y 1 ). in figure 4 , we show that indeed language models like gpt-2 produce poor predictions on incomplete subsets. ## connection to classification rationales in this section, we go over related rationalization approaches developed for classification and discuss why they cannot scale to sequence models. we also show that the combinatorial rationale objective in equation 3 is a global solution to a classification rationale-style objective. in classification problems, a sequence x 1:t is associated with a label y. rationale methods are commonly used in this setting @xcite @xcite @xcite . the most common approach uses two models: one, a selection model q(s|x 1:t ), provides a distribution over possible rationales; the other, the predictive model p(y|x s ), makes predictions using only samples from the former model. typically, p and q are both optimized to maximize here, f is the ground truth, unknown data distribution, and λ is a regularizing penalty that encourages smaller rationales. in practice, it is infeasible to adopt this objective for sequence models. equation 7 is centered on providing classification models with only the words in a sequence's rationale. in sequential settings, each word has a different rationale. since sequence models make t predictions per sequence and are trained by sharing all t word representations, each token would be indirectly exposed to words in the rationales of the words it is allowed to use. a remedy would be to train sequence models without sharing representations, but this is computationally infeasible; it requires o(t 3 ) computations per sequence for transformer architectures. most classification rationale methods treat q(s|x 1:t ) as a probability distribution over all possible rationales. however, the q that maximizes equation 7 is deterministic for any p. to see this, note that q does not appear inside the expectation in equation 7, so it can place all its mass on a single mode. we provide a formal justification in appendix b. since the optimal selection model q is a pointmass, the optimal rationale can be written as this optimization is identical to the combinatorial optimization in equation 3, albeit with a soft constraint on the rationale's prediction: the true label y is not required to be the maximum of p(y |x s ). in practice, this soft constraint sometimes results in empty rationales @xcite . since we view sufficiency as a key component of a good rationale, equation 3 imposes a hard constraint on the rationale's prediction. ## related work finding rationales is similar to feature selection. while global feature selection has been a wellstudied problem in statistics @xcite , instance-wise feature selection -where the goal is selecting features per-example -is a newer research area @xcite . we review local explanation methods used for nlp. gradient saliency. gradient-based saliency methods have long been used as a measure of feature importance in machine learning @xcite @xcite . some variations involve word embeddings @xcite ; integrated gradients, to improve sensitivity @xcite ; and relevance-propagation to track each input's contribution through the network @xcite . but there are drawbacks to using gradient-based methods as explanatory tools. @xcite show that in practice, gradients are saturated: they may all be close to zero for a wellfitted function, and thus not reflect importance. adversarial methods can also distort gradient-based saliences while keeping a model's prediction the same @xcite . we compare greedy rationalization to gradient saliency methods in section 8. ## experimental setup there are two goals in our empirical studies. the first is to compare the ability of greedy rationalization to other approaches for optimizing the combi-natorial objective in equation 3. the second is to assess the quality of produced rationales. we measure the quality of rationales using two criteria: faithfulness and plausibility. an explanation is faithful if it accurately depicts how a model makes a decision @xcite ); an explanation is plausible if it can be understood and interpreted by humans @xcite . although sufficiency is a standard way to measure faithfulness @xcite , all the rationales that satisfy the constraint of equation 3 are sufficient by definition. to measure plausibility, we compare rationales to human annotations. since there do not exist language modeling datasets with human rationales, we collected annotations based on lambada @xcite . the annotated dataset is available online, along with the code used for all experiments. 4 we compare greedy rationalization to a variety of gradient-and attention-based baselines (see section 6). to form baseline sequential rationales, we add words by the order prescribed by each approach, stopping when the model prediction is sufficient. the baselines are: l 2 gradient norms of embeddings @xcite , embedding gradients multiplied by the embeddings @xcite , integrated gradients @xcite , attention rollout (abnar and zuidema, 2020), the last-layer transformer attention weights averagedacross heads, and all transformer attentions averaged across all layers and heads @xcite . to compare rationale sets produced by each method to those annotated by humans, we use the set-similarity metrics described in deyoung et al. (2020): the intersection-over-union (iou) of each rationale and the human rationale, along with the token-level f1, treating tokens as binary predictions (either in the human rationale or out of it). we use transformer-based models for all of the experiments. we fine-tune each model for compatibility using a single gpu. that we can fine-tune gpt-2 large @xcite to learn compatible conditional distributions on a single gpu suggests that most practitioners will be able to train compatible models using a reasonable amount of computation. for model and fine-tuning details, refer to appendix d. ## results and discussion the experiments test sequential rationales for language modeling and machine translation. appendix e contains full details for each experiment. ## conclusion we proposed an optimization-based algorithm for rationalizing sequence predictions. although exact optimization is intractable, we developed a greedy approach that efficiently finds good rationales. moreover, we showed that models can be fine-tuned to form compatible distributions, thereby circumventing an intractable marginalization step. in experiments, we showed that the greedy algorithm is effective at optimization, and that its rationales are more faithful and plausible than those of gradient-and attention-based methods. we hope that our research, along with the release of an annotated dataset of sequence rationales, catalyzes further research into this area.