D ecoder L ens: Layerwise Interpretation of Encoder-Decoder Transformers (2024)

Abstract: In recent years, several interpretability methods have been proposed to interpret the inner workings of Transformer models at different levels of precision and complexity.In this work, we propose a simple but effective technique to analyze encoder-decoder Transformers. Our method, which we name DecoderLens, allows the decoder to cross-attend representations of intermediate encoder activations instead of using the default final encoder output.The method thus maps uninterpretable intermediate vector representations to human-interpretable sequences of words or symbols, shedding new light on the information flow in this popular but understudied class of models.We apply DecoderLens to question answering, logical reasoning, speech recognition and machine translation models, finding that simpler subtasks are solved with high precision by low and intermediate encoder layers.

## introduction many methods for interpreting the internal states of neural language models -and in particular transformer-based models -have been proposed in the last few years (for a review, see @xcite . such methods operate at many different levels of granularity, ranging from model-agnostic attribution methods that treat models as black-boxes, to probing methods that assess whether specific information is decodable from model representations, to fine-grained techniques aiming to causally link highly localized circuits to model behavior. these latter techniques (often referred to as 'mechanistic interpretability', @xcite are often strongly tied to model-specific components, and are likely to provide more faithful insight into how these models operate. what is the capital of spain? figure 1 : schematic overview of the decoderlens. by using the decoder to cross-attend intermediate encoder activation, we can gain qualitative insights into how representations evolve across encoder layers. we evaluate decoderlens empirically on a wide range of tasks, models, and domains. first, we demonstrate how representations evolve in flan-t5 @xcite by prompting the model to predict country capitals. next, we conduct an experiment in a more controlled domain, examining how transformers are able to resolve variable assignment in propositional logic. the restricted output space for this task allows us to closely inspect the kinds of solutions intermediate layers produce. finally, we apply the decoderlens to two common applications of encoder-decoder models: neural machine translation @xcite and speech-to-text transcription and translation @xcite . we find that intermediate outputs can be useful to find hypotheses about the strategies a model uses for solving (sub)tasks. one surprising finding, for example, is that flan-t5 encodes geographical information better in intermediate layers than in the top layer. additionally, our findings show that the middle encoder layers approximate correct transcriptions and translations well for models such as whisper and nllb. experiments from both logic and machine translation show that earlier layers sometimes output local approximations to their respective tasks. the decoderlens thus provides a useful tool that can be used in combination with other interpretability methods to gain a more complete insight into the inner workings of deep encoder-decoder language models. ## related work the current state of interpretability methods can be categorized by the different levels of granularity at which they explain model behavior. at the coarsest level, model-agnostic methods such as feature attributions (e.g., @xcite focus on explaining model output in terms of the most important input features. a major concern with this line of work is the faithfulness of a method: whether the attributions the method produces in fact correspond to the true, underlying causes of the model's output. the strong disagreement between different attribution methods raises doubts that the faithfulness requirement is met in practice @xcite @xcite . in response to these concerns, a novel line of work that has received increasing attention in recent years attempts to explain models at a more fine-grained level, leveraging knowledge about a model's inner workings based on specific components (e.g., @xcite @xcite . ## decoderlens the decoderlens approach is inspired by the log-itlens method of nostalgebraist (2020). the main intuition behind this method is that the residual stream in transformer decoder-only models forces representations across layers to gradually converge towards the final representation, iteratively refining its guess @xcite . this gradual change allows us to inspect how model predictions change across layers by directly applying the final unembedding transformation to intermediate hidden states. for encoder-decoder models, the logitlens can only be applied to the decoder component since there is no residual stream between encoder and decoder modules. to investigate how representations in the encoder evolve across layers, we therefore introduce the decoderlens, which leverages the entire decoder to verbalize the knowledge captured by intermediate encoder layers. this is achieved by early exiting the encoder at earlier layers, and using the resulting representations for the decoder crossattention operation. the decoderlens allows for richer insights than the logitlens, enabling the generation of full outputs from intermediate encoder states. it also may help mitigate out-of-distribution issues that can arise from using a single vocabulary projection (e.g. @xcite . the model outputs plausible strings that adhere to the original training objective, allowing us to see how the task is progressively addressed throughout encoder layers. we define the decoderlens as follows. for an encoder-decoder model m with n layers, the output of the decoder is normally generated based on the top-layer representations of the encoder, combined with a decoding algorithm (e.g. beam search). often, the encoder layers are first passed through a non-linear operation, such as layer normalization @xcite . the decoderlens operates similarly, by first passing the i th encoder layer through the non-linear operation f , and then feeding it as input to the decoder: in the following sections, we investigate the effectiveness of decoderlens by applying it to a variety of tasks, models, and domains. ## factual trivia qa we first apply the decoderlens to investigate the factual knowledge of a instruction-tuned encoderdecoder lm, flan-t5 (chung et al., 2022) foot_1 . as a case study, we consider country capital prediction, using prompts of the form "what is the capital of x?" and testing encoder layers' ability to produce correct outputs for all 193 united nations member states. we evaluate flan-t5 models of three sizes (large, xl, xxl, with 0.78b, 3.0b and 11.3b parameters respectively), containing the same number of layers (24) and hidden state size (1024), but differing in the feed-forward layer size and the number of attention heads @xcite . evaluation to investigate the types of responses generated by the decoderlens, we categorize model answers as follows: 1) correct response, based on a full string match, 2) incorrect response in the form of a different city name, 3) country name itself, 4) repetition of the question, 5) tautologies (the capital of x is the capital of x), 6) empty responses containing no alphanumeric characters, and 7) a miscellaneous category for anything that doesn't fall under these previous six categories. these categories were defined after a manual inspection of the decoderlens results: some examples of intermediate outputs can be seen in ## propositional logic results from the previous section indicate that de-coderlens can be useful for identifying the layers in which factual information arises and can be readily decoded in general pre-trained language models. in this section, we go one step further and apply decoderlens to a model exclusively trained on a downstream task. we believe it is advantageous to test novel interpretability methods on models that are trained to solve a simple, unambiguous task within a carefully controlled setup @xcite @xcite . we apply the decoderlens to a small transformer model that is trained from scratch on a synthetic (but non-trivial) task: predicting variable assignments given a logical formula. task we study an encoder-decoder model that is specifically trained on propositional logic, based on the setup of @xcite . the model is trained to output a partial satisfying assignment given a satisfiable formula in propositional logic. the models are trained in a standard sequence-tosequence setup using teacher forcing and only have access to a single correct output, even when several partial assignments would be semantically correct. nevertheless, this limited setup seems sufficient to teach these models the semantics of propositional logic @xcite . at test time, the models are able to output novel assignments to unseen formulas with 93% accuracy. ## machine translation we apply decoderlens to nllb-600m @xcite , a state-of-the-art multilingual model trained in over 200 languages, to quantify encoder influence on translation quality and properties. we use 1012 sentences from the dev/test split of flores-101 @xcite , using english ↔ {italian, french, dutch} as high-resource pairs and english ↔ {xhosa, zulu} as low-resource pairs to evaluate differences in intermediate encoder layers' performances for these two settings. ## speech-to-text we next apply decoderlens to whisper @xcite , a state-of-the-art multilingual speech model trained on a set of supervised audio-to-text tasks, including multilingual speech transcription and speech translation to english. we use whisper in three different sizes (base, small, and medium) which differ in their number of layers (6, 12, and 24, respectively). data we use covost 2 @xcite , a multilingual speech-to-text translation dataset based on common voice corpus @xcite . we sample 100 sentences for nine languages: english (en), french (fr), spanish (es), portuguese (pt), dutch (nl), japanese (ja), arabic (ar), persian (fa), and turkish (tr). since the dataset includes both source and translation references for each utterance, we can inspect whisper's behavior for both transcription and translation tasks on the same examples, providing an unbiased comparison between the tasks. results figure 6 shows the overall results of word error rate (wer) across various source languages when applying decoderlens to whispermedium for transcription and translation tasks. while the overall pattern of wer is decreasing, we can discern that fundamental information emerges from the intermediate layers. comparing the trend of wer for transcription and translation, it appears that the essential information required for transcription is encoded in earlier encoder layers compared to translation. 6 ## conclusion our work contributes to a growing body of research on the interpretability of language models. by in-troducing the decoderlens, we provide insights into how intermediate encoder representations of encoder-decoder transformers influence decoder predictions. we apply our method to various models and tasks, finding that intermediate outputs can provide valuable insights into the model's decisionmaking process. in particular, our findings reveal how "simpler" subtasks (e.g., simple conjunctive logic formulas, high-resource mt, speech transcription) are captured by early encoder layers with high precision and persist up to the final model output through the residual stream, while more challenging tasks (e.g., complex logic formulas, low-resource mt, speech translation) are only addressed by final encoder layers. we also find some evidence that early layer outputs are more local solutions. errors in variable assignments in the middle layer of the transformer trained on propositional logic are due to the model solving subparts of the sentence independently. additionally, translations from earlier layers of nllb adhere more closely to the word order of the input. these observations are in line with previous work on probing, which showed that linguistic subtasks in lms are performed at different stages in transformers @xcite . moreover, it provides evidence that model predictions are refined iteratively also across encoder layers, complementing previous work on decoder-only models. by verbalizing the knowledge encoded in intermediate model layers, decoderlens can provide useful and humaninterpretable insights into the evolution of model predictions, complementing other interpretability techniques for the study of neural language models. future work could explore the application of de-coderlens to the universal transformer @xcite , especially for algorithmic tasks @xcite where its intermediate representation might be more interpretable and compositional thanks to weight sharing. moreover, the tendency of earlier layers to produce simpler generations can be connected to outputs produced during early stages of model training @xcite . in this context, decoderlens might be used to investigate the relation between training dynamics and information geometry across model layers @xcite . lastly, decoder-lens could be used as diagnostic tool to investigate where wrong model predictions emerge, which is useful for both interpretability purposes and model improvement through early exiting strategies.