Measuring the Mixing of Contextual Information in the Transformer (2022)

Abstract: The Transformer architecture aggregates input information through the self-attention mechanism, but there is no clear understanding of how this information is mixed across the entire model. Additionally, recent works have demonstrated that attention weights alone are not enough to describe the flow of information. In this paper, we consider the whole attention block –multi-head attention, residual connection, and layer normalization– and define a metric to measure token-to-token interactions within each layer. Then, we aggregate layer-wise interpretations to provide input attribution scores for model predictions. Experimentally, we show that our method, ALTI (Aggregation of Layer-wise Token-to-token Interactions), provides more faithful explanations and increased robustness than gradient-based methods.

## introduction the transformer @xcite has become ubiquitous in different tasks across multiple domains, becoming the architecture of choice for many nlp @xcite and computer vision @xcite tasks. the self-attention mechanism inside the transformer is in charge of combining contextual information in its intermediate token representations. attention weights offer a straightforward layer-wise interpretation, as they provide a distribution over input units, which is often presented as giving the relative importance of each input. a prominent line of research has investigated the faithfulness of attention weights @xcite @xcite @xcite with contradictory conclusions. some works have studied layer-wise attention patterns by analyzing standard attention @xcite @xcite and effective grad ℓ2 went here just before a movie . the service was fast but that ' s it . i ordered the mango and shrimp quesadilla . my friend ordered nachos . the food was not good . i and my friend could not finish our food and we had stomach aches immediately . ig ℓ2 went here just before a movie . the service was fast but that ' s it . i ordered the mango and shrimp quesadilla . my friend ordered nachos . the food was not good . i and my friend could not finish our food and we had stomach aches immediately . alti went here just before a movie . the service was fast but that ' s it . i ordered the mango and shrimp quesadilla . my friend ordered nachos . the food was not good . i and my friend could not finish our food and we had stomach aches immediately . in this work, we propose alti, an interpretability method that provides input tokens relevancies foot_0 to the model predictions by measuring the aggregation of contextual information across layers. we use the attention block decomposition proposed by @xcite and refine the measure of the contribution of each input token representation to the attention block output (layer-wise token-totoken interactions), based on the properties of the representation space and the limitations of previously proposed metrics. we then aggregate the layer-wise explanations and track the mixing of information in each token representation, yielding input attributions for the model predictions. finally, in the text classification and subject-verb agreement tasks, we show alti scores higher than gradient-based methods and previous similar approaches in two common faithfulness metrics, while showing greater robustness. the code to reproduce the experiments is publicly available. foot_1 2 background ## experimental setup we perform our experiments in the text classification (tc) and the subject-verb agreement (sva) tasks. the former evaluates how models classify an entire input sequence, the latter assesses the ability of a model to capture syntactic phenomena @xcite . for the tc task, we use the stanford sentiment treebank v2 (sst-2) @xcite composed of short sentences, imdb @xcite with movies reviews and longer inputs than sst2, and yelp dataset challenge @xcite containing user's reviews from businesses of similar length than in imdb. all of them have positive and negative sentiment sentences. for the sva task, we use linzen et al. (2016) dataset, which includes sentences from wikipedia containing a present-tense verb that agrees in grammatical number (singular/plural) with the head of the subject. the sentence is fed into the model with its verb masked, and the model is asked to predict if the masked verb is singular or plural foot_4 (binary classification): at least four players from the 1983 draft now [mask] plural as coaches. ## results in this section, we present quantitative and qualitative results comparing alti with other input attribution methods. ## conclusions in this paper, we have presented alti, an input attribution method that quantifies the mixing of information in the transformer. we have demonstrated that with accurate layer-wise token-to-token contribution measurements relying on ℓ 1 -based metrics, the interpretable attention decomposition of the attention block is a powerful tool when combined with the rollout method. empirically, we show that alti outperforms every input attribution method we have experimented with in two common faithfulness metrics, while showing greater robustness. overall, we believe this opens new possibilities for studying contextual information aggregation across the transformer.