Explaining How Transformers Use Context to Build Predictions (2023)

Abstract: Language Generation Models produce words based on the previous context. Although existing methods offer input attributions as explanations for a model’s prediction, it is still unclear how prior words affect the model’s decision throughout the layers. In this work, we leverage recent advances in explainability of the Transformer and present a procedure to analyze models for language generation. Using contrastive examples, we compare the alignment of our explanations with evidence of the linguistic phenomena, and show that our method consistently aligns better than gradient-based and perturbation-based baselines. Then, we investigate the role of MLPs inside the Transformer and show that they learn features that help the model predict words that are grammatically acceptable. Lastly, we apply our method to Neural Machine Translation models, and demonstrate that they generate human-like source-target alignments for building predictions.

## introduction language generation models, like transformerbased language models @xcite have recently revolutionized the field of natural language processing (nlp). despite this, there is still a gap in our understanding of how they are able to produce language that closely resembles that of humans. this means that we are unable to determine the cause of a model's failure in specific instances, which can result in the generation of hallucinated content or toxic output. the majority of previous work in explainability of nlp model predictions has focused on analyzing them on downstream tasks, generally with a small output space, such as text classification or natural language inference @xcite @xcite . this line of research includes a large body of work focusing on the analysis of the attention mechanism recently, several works have tackled the interpretability of transformers @xcite on the language modeling task. @xcite studied the transformer from the residual stream perspective, depicted in figure 1 , where different components (mlps, attention heads...) read and write to subspaces of the residual stream. this approach has aided in explaining certain behaviours of language models, like induction heads @xcite , where attention heads search over the context for previous repetitions of the same token and copy the next token, or even specialized heads solving the indirect object identification (ioi) task @xcite . similarly, mlps inside the transformer have also been studied as elements writing into the residual stream. @xcite observed that mlp blocks can act as key-value memories, where values add to the residual, thus promoting the prediction of words that convey similar semantic meaning. furthermore, the attention mechanism in the transformer, composed of attention heads, an output weight matrix, and a layer normalization, can be decomposed into an interpretable operation @xcite , providing layerwise explanations which have proven to be highly faithful @xcite . in this work, we propose explaining the predictions of transformers language generators by combining the residual stream analysis perspective with the attention decomposition. our approach measures the amount of logit (pre-activation of the softmax) added or subtracted by each token representation at each layer. we then track the logit contributions back to the model's input by aggregating across layers (logit explanation). additionally, we consider the mixing of information in intermediate layers by using alti @xcite (alti-logit explanation). to evaluate the proposed interpretability methods, we follow the recently introduced contrastive explanations framework @xcite , which aims to explain why the model predicted one token instead of a foil token, a priori explained by some linguistic phenomena evidence. then, we analyze the role of mlps and show that they aid the model in determining predictions that follow grammar rules. finally, we demonstrate that nmt models generate human-like source-target alignments for building translations. foot_0 2 approach ## experimental setup we evaluate the quality of our proposed method through contrastive explanations. following yin and neubig (2022) we use a subset of blimp dataset @xcite , which contains sentence pairs with small variations in grammatical phenomena id example (acceptable/unacceptable) anaphor agreement aga karla could listen to herself/himself. ana eva approached herself/themselves. argument structure asp gerald is hated by the teachers/pie. determiner-noun agreement dna eva has scared these children/child. dnai tammy was observing that man/men. dnaa the driver sees that unlucky person/people. dnaai phillip liked that smooth horse/horses. npi licensing npi even danielle also/ever leaves. subject-verb agreement darn the grandfathers of diana drink/drinks. ipsv many people have/has hidden away. rpsv most associations buy/buys those libraries. for each linguistic phenomena, we use spacy @xcite and follow @xcite rules to find the evidence (in previous tokens), that is enforcing grammatical acceptability ( we also add to the analysis sva (subject-verb agreement) @xcite and the indirect object identification (ioi) @xcite datasets. the sva dataset includes nouns with an opposite number to that of the main 6 blimp ids. aga: anaphor_gender_agreement; ana: anaphor_number_agreement; asp: animate_subject_passive; dna: we use gpt-2 xl (1.5b) model @xcite , as in @xcite , as well as other autoregressive transformer language models, such as gpt-2 small (124m), and gpt-2 large models (774m), opt 125m @xcite , and bloom's 560m and 1.1b variants (bigscience workshop, 2022), through hugging-face library @xcite . alignment metrics. following yin and neubig (2022), we define the evidence as a binary vector @xmath0, with all zeros except in the position of the tokens inside the evidence, i.e. the tokens which the prediction depends on, extracted by the rule. explanations are vectors, @xmath1. to measure the alignment between an explanation and the evidence we use mrr (mean reciprocal analysis). sorting the tokens in descending order, mrr evaluates the average of the inverse of the rank of the first token that is part of b. although @xcite use also dot-product and probes needed metrics for measuring alignments, dot-product favors grad norm explanations since it gives positive scores only, and probes needed is closely related to mrr, giving redundant results. ## contrastive methods yin and neubig (2022) proposed extending different common input attribution methods to the contrastive setting. in §5 we compare their explanations with the ones obtained with our proposed contrastive methods (equation ( 12 )). ## analysis of mlps the mlp block in the transformer contains two learnable weight matrices foot_4 : w l 1 ∈ r d×d mlp and w l 2 ∈ r d mlp ×d , and an element-wise non-linear activation function α. it takes as input the state of the residual stream at timestep t ( xl t ) and computes: following, o l t is added back to the residual stream (figure 1 ). equation ( 16 ) can be seen as key-value memories @xcite , where keys are stored in components of k @xmath2, and values (v l ) are rows of w 2 . following the keyvalue perspective, equation ( 16 ) can be rewritten as thus, the update of the mlp can be decomposed into sub-updates @xcite performed by each the update in the logit's difference between the target and foil tokens by each value i is therefore: ## neural machine translation an nmt system estimates the likelihood of a target sequence of tokens, @xmath3, . . . , y t ), given a source sequence of tokens, @xmath4, . . . , x i ): cross-attention. similar to equation ( 6 ), the output of the cross-attention ( y c,l t ) and self-attention ( y s,l t ) (figure 9 ) of a decoder layer in an encoderdecoder transformer can be decomposed foot_5 as as shown in figure 9 , each transformed vector updates the logits of the token predictions by multiplying it with the corresponding column of u , as in equation ( 8 ): ## related work the projection of lms representations and model parameters to the vocabulary space has been a subject of previous research @xcite . @xcite view feedforward layers as performing updates to the probability distribution of the token predictions. mickus et al. (2022) study how the different transformer modules contribute to the hidden representations, and dar et al. (2022) directly interpret transformer static parameters in the embedding space. in this work, our focus lies in interpreting the influence of input tokens and its representations in the model predictions. furthermore, work on mechanistic interpretability @xcite has discovered 'circuits' within lms in charge of solving tasks @xcite . in contrast to their methods, our approach does not rely on causal interventions in the computations of transformers. more broadly, our work can be related to those explaining the prediction process of lms @xcite @xcite . ## conclusions in this paper, we introduce a new procedure for analyzing language generation models by combining the residual stream perspective with interpretable attention decomposition, and tested our approach using contrastive examples in transformer lms. we found that the explanations provided by our proposed methods, logit and alti-logit, align better with available linguistic evidence in the context of the sentence, compared to common gradientbased and erasure-based baselines. we also analyzed the role of mlps and showed that they assist the model in determining predictions that conform to the grammar rules. additionally, we applied our method to a machine translation model and demonstrated that it generates human-like alignments for building predictions. overall, our results suggest that decomposing the logit scores is an effective way to analyze language generation models.