O N I DENTIFIABILITY IN T RANSFORMERS
Anonymous authors
Paper under double-blind review
A BSTRACT
In this work we delve deep in the Transformer architecture by investigating two of its core components: self-attention and contextual embeddings.
In particular, we study the identiﬁability of attention weights and token embeddings, and the aggregation of context into hidden tokens.
We show that attention weights are not unique and propose effective attention as a complementary tool for improving explanatory interpretations based on attention.
Furthermore, we are the ﬁrst to show that input tokens retain to a large degree their identity across the model. We also study how identity information propagates and ﬁnd that it is mainly encoded in the angle of the embeddings and gradually decreases with depth.
Finally, we demonstrate strong mixing of input information in the generation of contextual embeddings by means of a novel quantiﬁcation method based on gradient attribution.
Overall, we show that self-attention distributions are not directly interpretable and present tools to further investigate Transformer models.
I NTRODUCTION
In this paper we investigate neural models of language based on self-attention by leveraging the concept of identiﬁability .
Intuitively, identiﬁability refers to the ability of a model to learn stable representations.
This is arguably a desirable property, as it affects the replicability and interpretability of the model’s predictions.
Concretely, we focus on two aspects of identiﬁability.
The ﬁrst is related to structural identiﬁability (Bellman & ˚Astr¨om, 1970): the theoretical possibility (a priori) to learn a unique optimal parameterization of a statistical model.
From this perspective, we analyze the identiﬁability of attention weights, what we call attention identiﬁability , in the self-attention components of transformers (Vaswani et al., 2017), one of the most popular neural architectures for language encoding and decoding.
We also investigate token identiﬁability as the ﬁne-grained, word-level mappings between input and output generated by the model.
The role of attention as a means of recovering input-output mappings, and various types of explanatory insights, is currently the focus of much research and depends to a signiﬁcant extent on both types of identiﬁability.
We contribute the following ﬁndings to the ongoing work: With respect to attention indentiﬁability, in Section 3 we show that – under mild conditions with respect to input sequence length and attention head dimension – the attention weights for a given input are not identiﬁable.
This implies that there can be many different attention weights that yield the same output.
This ﬁnding challenges the direct interpretability of attention distributions.
As a supplement, we propose the concept of effective attention , a diagnostic tool that examines attention weights for model explanations by removing the weight components that do not inﬂuence the model’s predictions.
With respect to token identiﬁability, in Section 4, we devise an experimental setting where we probe the hypothesis that contextual word embeddings maintain their identity as they pass through successive layers of a transformer.
This is an assumption made in much current research, which has not received a clear validation yet.
Our ﬁndings give substance to this assumption, although it does not always hold in the last layers.
Furthermore, we show that the identity information is largely encoded in the angle of the embeddings and that it propagates in a linear manner.
In Section 5 we further investigate the contribution of all input tokens in the generation of the contextual embeddings in order to quantify the mixing of token and context information.
We introduce Hidden Token Attribution , a quantiﬁcation method based on gradient attribution.
We ﬁnd that selfattention strongly mixes context and token contributions.
Token contribution decreases monotonically with depth, but the corresponding token typically remains the largest individual contributor.
We also ﬁnd that, despite visible effects of long term dependencies, the context aggregated into the hidden embeddings is mostly local.
We notice how, remarkably, this must be an effect of learning.
B ACKGROUND ON T RANSFORMERS
The Transformer (Vaswani et al., 2017) is the neural architecture of choice for natural language processing (NLP).
At its core it consists of several multi-head self-attention layers.
In these, every token of the input sequence attends to all other tokens by projecting its embedding to a query, key and value vector.
Formally, let Q ∈ R d s × d q be the query matrix, K ∈ R d s × d q the key matrix and
V ∈ R d s × d v the value matrix, where d s is the sequence length and d q and d v the dimension of the query and the value vectors, respectively.
The output of an attention head is given by:
[Equation]
The attention matrix A ∈ R d s × d s calculates for each token in the sequence how much the hidden embedding at this sequence position attends to each of the other (hidden) embeddings.
Self-attention is a non-local operator, which means that at any layer a token can attend to all other tokens regardless of the distance in the input.
Self-attention thus produces so-called contextual word embeddings , as successive layers gradually aggregate contextual information into the embedding of the input word.
We focus on a Transformer model called BERT (Devlin et al., 2019), although our analysis can be easily extended to other models such as GPT (Radford et al., 2018; 2019).
BERT operates on input sequences of length d s .
We denote input tokens in the sentence as x i , where i ∈ [1 , ..., d s ] .
We use x i ∈ R d with embedding dimension d to refer to the sum of the token-, segment- and position embeddings corresponding to the input word at position i .
We denote the contextual embedding at position i and layer l as e li .
Lastly, we refer to the inputs and embeddings of all sequence positions as matrices X and E , respectively, both in R d s × d .
For all experiments we use the pre-trained uncased
BERT-Base model as provided by Devlin et al.
A TTENTION I DENTIFIABILITY
We begin with the identiﬁability analysis of self-attention weights.
Drawing an analogy with structural identiﬁability (Bellman & ˚Astr ¨ om, 1970), we state that the attention weights of an attention head for a given input are identiﬁable if they can be uniquely determined from the head’s output.
We emphasize that attention weights are input dependent and not model parameters .
However, their identiﬁability affects the interpretability of the output (cf. Jain & Wallace (2019) and Wiegreffe & Pinter (2019)).
If attention is not identiﬁable, explanations based on attention may be unwarranted.
The output of a multi-head attention layer is the summation over each of the h single head outputs (cf.
Eq. 1) multiplied by the matrix H ∈
R d v × d with reduced head dimension d v = d/h ,
[Equation]
where W V ∈ R d × d v projects the embedding E into the value matrix V = EW V , and we deﬁne
T = EW V H .
Here, the layer and head indices are omitted for simplicity, since the proof below is valid for each individual head and layer in Transformer models.
Intuitively, the head output is a linear combination of T vectors using the attention as weighting coefﬁcient.
If the sequence length, i.e. the number of weighting coefﬁcient, is larger than the rank of T , attention weights are not uniquely determined from the head output; i.e., they include free variables.
In other words, some of the T vectors are linear combinations of others, the pivots.
We now prove, by analyzing the null space dimension of T , that attention weights are not identiﬁable using the head or ﬁnal model output.
3. U PPER B OUND FOR RANK ( T )
We ﬁrst derive the upper bound of the rank of matrix T = EW V H .
Note that rank ( ABC ) ≤ min ( rank ( A ) , rank ( B ) , rank ( C )) , therefore,
[Equation]
The second step holds since rank ( E ) ≤ min ( d s , d ) , rank ( W V ) ≤ min ( d, d v ) and rank ( H ) ≤ min ( d v , d ) .
3. T HE NULL SPACE OF T
The null space of T describes all vectors that are mapped to vector 0 by T :
[Equation]
Its special property is that, for ˜ A =
[˜ x 1 , ˜ x 2 , ..., ˜ x
d s ] T where ˜ x T i is any vector in this null space,
[Equation]
This implies that there are inﬁnitely many different attention weights A + ˜ A that lead to the exact same attention layer output and ﬁnal model outputs, if the dimension of the null space is not zero.
Due to the Rank Nullity theorem, the dimension of the null space of T is [Equation] otherwise
[Equation]
where equality holds if E , W V and H are of full rank and their matrix product does not bring further rank reductions.
Hence, when the sequence length is larger than the attention head dimension ( d s > d v ), self-attention is non-identiﬁable.
Furthermore, the null space dimension increases with the sequence length.
3. E FFECTIVE ATTENTION
The non-identiﬁability of self-attention, due to the existence of the non-trivial null space of T , challenges the interpretability of attention weights.
However, one can decompose attention weights
A into the component in the null space A (cid:107) and the component orthogonal to the null space
A ⊥ :
[Equation]
since A (cid:107) ∈ null ( T ) = ⇒ A (cid:107) T = 0 .
Hence, we propose a novel concept named effective attention ,
[Equation]
which is the part of the attention weights that actually have an effect on the model output.
Effective attention can serve as a complementary diagnostic tool for examining how attention weights inﬂuence model outputs.
To illustrate the point, Figure 1 compares (a) raw attention A and (b) effective attention A ⊥ .
Using the same Wikipedia samples as in Clark et al.
with maximum sequence length 128, we compute the average attention of BERT and compare it to the corresponding average effective attention.
Clark et al.
(2019) conclude that the [CLS] token attracts more attention in early layers, the [SEP] tokens attract more in middle layers, and periods and commas do so in deep layers.
However, combined with a gradient based investigation, they suggests this attention to the [SEP] token is generally a ’no-op’. The effective attention weights suggest a more consistent pattern: while periods and commas seem to generally attract more attention than [CLS] and [SEP], the peak of [SEP] token observed by raw attention has disappeared.
This shows that the peak of [SEP] token is irrelevant to the computation of the output for middle layers.
Furthermore, the effective attention can suggest if some attention on [SEP] may have some effect on the head outputs. The same arguments also hold for the sharp peak of raw attention on punctuation tokens between
Layer
R a w
A tt e n ti on (a)   0 .0 .0 .0 .
Layer
E ff ec ti v e A tt e n ti on [CLS] [SEP].or, (b)  0 .0 .0 .
Seq.
Length
P ea r s on ’ s r (c) layers 10 and 12.
An additional example showing similar results can be found in Appendix A.2.
See also Appendix A.3, where we discuss in depth an example where effective attention would support interpretive conclusions that differ from those one might draw solely based on raw attention.
Thus, effective attention can help discover interesting interactions encoded in the attention weights which may be otherwise obfuscated by the null attention. Finally, we show that effective attention can diverge signiﬁcantly from raw attention. We note that the Pearson correlation between effective and raw attention decreases with sequence length as shown in Figure 1c. This is in line with our theoretical ﬁnding in Eq. 6 that states an increase in the dimension of the null space with the sequence length.
Given a bigger null space, more of the raw attention becomes irrelevant, yielding a lower correlation between effective and raw attention.
Note also that for sequences with fewer than d v = 64 tokens, the associated null space dimension is zero, and hence attention and effective attention are identical (Pearson correlation of value 1).
This loss of correlation with increased sequence length questions the use of attention as explanation in practical models, where it is not uncommon to use large sequence lengths.
A few examples include: BERT for question answering (Alberti et al., 2019) and XL-Net (Yang et al., 2019) with d s = 512 , or document translation (Junczys-Dowmunt, 2019) with d s = 1000 .
T OKEN I DENTIFIABILITY
We now study the other fundamental element of transformers; the internal vector representations of tokens, or contextual word embeddings.
It is commonly assumed that a contextual word embedding keeps its “identity”, which is tied to the input word, as it passes through the self-attention layers.
Speciﬁcally, we identify three cases where this assumption is made implicitly without justiﬁcation.
First, visualizations/interpretations linking attention weights to attention between words, when in fact the attention is between embeddings, i.e., mixtures of multiple words (Vaswani et al., 2017; Devlin et al., 2019; Vig, 2019; Clark et al., 2019; Raganato & Tiedemann, 2018; Voita et al., 2019; Tang et al., 2018; Wangperawong, 2018; Padigela et al., 2019; Baan et al., 2019; Dehghani et al., 2019;
Zenkel et al., 2019).
Second, accumulation methods that sum the attention to a speciﬁc sequence position over layers and/or attention heads, when the given position might encode a different mixture of inputs in each layer (Clark et al., 2019; Baan et al., 2019; Klein & Nabi, 2019; Coenen et al., 2019).
Finally, using classiﬁers to probe hidden embeddings for word-speciﬁc aspects without factoring in how much the word is still represented (Lin et al., 2019; Peters et al., 2018).
To investigate this assumption we introduce the concept of token identiﬁability , as the existence of a one-to-one mapping between contextual embeddings and their corresponding input tokens.
Formally, a token e li is identiﬁable if there exists a function g ( · ) such that g ( e li )
=
x i .
We cannot prove the existence of g analytically.
Instead, for each layer l we use a function approximator ˆ g l ( e li )
=
ˆ x i trained on a dataset of ( e li , x i ) pairs.
We then search for the nearest neighbour ˆ x 1 nni of ˆ x i within the same sentence, and say that e lj is identiﬁable if ˆ x 1 nni
= x i .
For evaluation we then report the token identiﬁability rate deﬁned as the percentage of correctly identiﬁed tokens.
Layer
I d e n ti ﬁ a b ilit y
R a t e ˆ g MLPcos,l ˆ g lincos,l ˆ g naivecos,l ˆ g MLPl 2 ,l ˆ g linl 2 ,l ˆ g naivel 2 ,l (a)      0 .0 .0 .
Layer
I d e n ti ﬁ a b ilit y
R a t e l= l=6 l= l= (b) 4. S ETUP For the experiments in this and subsequent sections we use the development dataset from the Microsoft Research Paraphrase Corpus (MRPC) dataset (Dolan & Brockett, 2005), while in Appendix D we provide results on two additional datasets.
The MRPC development set contains 408 examples with a sequence length
d s between 26 and 92 tokens, with 58 tokens on average.
We pass allsentences (21,723 tokens) through BERT and extract for each token the input embeddings x i and the hidden embeddings e li at all layers.
We then train ˆ g on the regression task of predicting input tokens x
i from hidden tokens e li .
We experiment with two loss functions and measures for ﬁnding the nearest neighbour; cosine distance and L 2 distance.
We use 10-fold cross validation with 70/15/ train/validation/test splits per fold and ensure that tokens from the same sentence are not split across sets.
The validation set is used for early stopping.
See Appendix B.1 for further details.
4. E XPERIMENTAL R ESULTS AND D ISCUSSION
In a ﬁrst experiment, we use a linear perceptron without bias, i.e., ˆ g linl ( x ) = W x , and a non-linear MLP ˆ g MLPl , where training, validation and test data all come from layer l .
Figure 2a shows the test set token identiﬁability rate of ˆ g l for l = [1 , ..., 12] .
We also report a naive baseline ˆ g naivel ( e li ) =
e li , i.e., we directly retrieve the nearest neighbour of e li from the input tokens.
The results for ˆ g naivel show that, according to both distance measures, contextual embeddings in BERT stay close to their input embeddings up to layer 4, followed by a linear decrease in token identiﬁability rate. By training a transformation to recover the original embedding, we see that most of the identity information is still present in the contextualized embeddings.
Speciﬁcally, a linear projection is enough to recover 93% of the tokens in the last layer based on a cosine distance nearest neighbour lookup.
This experiment shows that although the identiﬁablity rate decreases with depth, tokens remain mostly identiﬁable across layers.
Furthermore, we ﬁnd that a linear transformation ˆ g linl that preserves the relative angles between embeddings is more effective to recover identity when cosine distance is used than an MLP with L 2 distance. Therefore, we conjecture that BERT encodes most of the identity information in the angle of the embeddings.
Finally, Lin et al.
(2019) show that BERT discards much of the positional information after layer 3.
However, tokens remain largely identiﬁable throughout the model, indicating that BERT does not only rely on the positional embeddings to track token identity.
To provide further insights into contextual word embeddings, Appendix B. shows results for recovering neighbouring input tokens.
In a second experiment we test how well the ˆ g lincos,l trained only on ( e li , x i ) pairs from one layer l generalizes to all layers, see Figure 2b.
For l = 1 , the token identiﬁability rate on subsequent layers drops quickly to below 70% at layers 11 and 12.
Interestingly, for l = 12 a very different pattern can be observed, where the identiﬁability is 94% for layer 12 and then almost monotonically increases when testing on earlier layers.
Further, for l = 6 we see both patterns. [Figure]
This experiment suggests that the nature of token identity changes as tokens pass through the model, and patterns learned on data from later layers transfer well to earlier layers.
The experiment also shows that layer 12 is behaving differently than the other layers.
In particular, generalizing to layer 12 from layer 11 seems to be difﬁcult, signiﬁed by a sudden drop in token identiﬁability rate.
We believe this is due to a task dependent parameter adaptation induced in the last layer by the nextsentence prediction task which only uses the CLS token (cf. Appendix B.4 for additional hints that the last layer(s) behave differently).
See Appendix B.3 for results of ˆ g linl 2 ,l , ˆ g MLPl 2 ,l and ˆ g MLPcos,l .
A TTRIBUTION A NALYSIS TO I DENTIFY C ONTEXT C ONTRIBUTION
We conclude by considering the role of the contextual information in the hidden embeddings.
We introduce Hidden Token Attribution , a context quantiﬁcation method based on gradient attribution (Simonyan et al., 2014) to investigate the hidden tokens’ sensitivity with respect to the input tokens.
5. H IDDEN T OKEN A TTRIBUTION
Gradient based attribution approximates the neural network function f ( X ) around a given sequence of input word embeddings
X ∈ R d s × d by the linear part of the Taylor expansion.
With this, the network sensitivity is analyzed by looking at how small changes at the input correlate with changes at the output.
Since in the linear approximation this change is given by the gradient ∇ x i f
= δf ( X ) δ x i for a change in the i -th input token x i ∈ R d of X , the attribution of how much input token x i affects the network output f ( X ) can be approximated by the L 2 norm of the gradient: attr ( x i ) =
||∇ x i f || 2 .
Since we are interested in how much a given hidden embedding e l j attributes to the input tokens x i , i ∈ [1 , 2 , . . .
, d s ] , we deﬁne the relative input contribution c li,j of input x
i to output f ( X ) =
e lj as
[Equation]
Since we normalize by dividing by the sum of the attribution values to all input tokens, we obtain values between 0 and 1 that represent the contribution of each input token
x i to the hidden embedding e lj .
Hidden Token Attribution differs from the standard use of gradient attribution in that, instead of taking the gradients of the output of the model with respect to the inputs in order to explain the model’s decision, we calculate the contribution of the inputs to intermediate embeddings in order to track the mixing of information.
Further details of this method are discussed in Appendix C.1.
5. T OKEN M IXING : C ONTRIBUTION OF I NPUT T OKENS
We use Hidden Token Attribution to extend the results of Section 4 showing how much of the input token is contained in a given hidden embedding.
In Figure 3a we report the contribution c
lj,j of input tokens x j to their corresponding hidden embeddings e l j at the same position j for each layer Layer R e l . C on t r i bu ti on ( % ) 1st
2nd 3rd 4thand5th 6thto10th 11thonwards (a) - - -      0 .
Neighbour T o t a l C on t r i bu ti on l =l =l =(b) l . After the ﬁrst layer the median contribution of the input token is less than a third (30.6%).
The contribution then decreases monotonically with depth; at layer 6 the median is only 14.4% and after the last layer it is 10.7%.
In Appendix C.5 we provide detailed results by word type.
Next, we study which input token is the largest contributor to a given hidden embedding e lj .
The corresponding input token x j generally has the largest contribution.
Figure 3b shows the percentage ˜ P of tokens that are not the highest contributor to their hidden embedding at each layer.
In the ﬁrst three layers the original input x j always contributes the most to the embedding e lj .
In subsequent layers, ˜ P increases monotonically, reaching 18% in the sixth layer and 30% in the last two layers.
These results show that, starting from layer three, self-attention strongly mixes the input information by aggregating the context into the hidden embeddings.
This is in line with the results from Section 4, where we see a decrease in token identiﬁability rate after layer three.
However, ˜ P is always higher than the token identiﬁability error at the same layer, indicating that tokens are mixed in a way that often permits recovering token identity even if the contribution of the original token is outweighed by other tokens.
This implies that there is some “identity information” that is preserved through the layers.
The strong mixing of information further questions the common assumption that attention distributions can be interpreted as “how much a word attends to another word”.
5. C ONTRIBUTION OF C ONTEXT TO H IDDEN T OKENS
In this section we study how context is aggregated into hidden embeddings.
Figure 4a shows the relative contribution of neighbouring tokens at each layer for the relative positions: ﬁrst, second, third, fourth and ﬁfth together, sixth to 10th together, and the rest.
The closest neighbours (1st) contribute signiﬁcantly more in the ﬁrst layers than in later layers.
Conversely, the most distant neighbours (11th onwards) contribute the most in deeper layers (cf.
Appendix C.2).
Despite the progressive increase in long-range dependencies, the context in the hidden embeddings remains mostly local.
Figure 4b represents the normalized total contribution aggregated over all tokens from each of their neighbours at the ﬁrst, middle and last layer.
This ﬁgure shows that the closest neighbours consistently contribute the most to the contextual word embedding regardless of depth.
On the other hand, we indeed observe an increase of distant contributions at later layers.
The results of this section imply that BERT learns local operators from data in an unsupervised manner, in the absence of any such prior in the architecture.
This behavior is not obvious, since attention is a highly non-local operator, and in turn indicates the importance of local dependencies in natural language.
While contribution is local on average , we ﬁnd that there are exceptions, such as the [CLS] token (cf.
Appendix C.3).
Furthermore, using our Hidden Token Attribution method, one can track how context is aggregated for speciﬁc tokens (cf.
Appendix C.4).
R ELATED W ORK
Input-output mappings play a key role in NLP.
E.g., in machine translation, they were introduced in the form of explicit alignments between source and target words (Brown et al., 1993).
Neural translation architectures re-introduced this concept in the form of attention (Bahdanau et al., 2015).
The development of multi-head self-attention (Vaswani et al., 2017) has led to many impressive results in NLP.
As a consequence, much work has been devoted to better understand what these models learn, with a particular focus on using attention to explain model decisions.
Jain & Wallace (2019) show that attention distributions of LSTM based encoder-decoder models are not unique, and that adversarial attention distributions that do not change the model’s decision can be constructed.
They further show that attention distributions only correlate weakly to moderately with dot-product based gradient attribution.
Wiegreffe & Pinter (2019) also ﬁnd that adversarial attention distributions can be easily found, but that these alternative distributions perform worse on a simple diagnostic task.
Serrano & Smith (2019) ﬁnd that zero-ing out attention weights based on gradient attribution changes the output of a multi-class prediction task more quickly than zeroing out based on attention weights, thus showing that attention is not the best predictor of learned feature importance. Concurrent work Pruthi et al. (2019) demonstrates that self-attention models can be manipulated to produce different attention masks with very little cost in accuracy.
These papers differ in their approaches, but they all provide empirical evidence showing that attention distributions are not unique with respect to downstream parts of the model (e.g., output) and hence should be interpreted with care.
Here, we support these empirical ﬁndings by presenting a theoretical proof of the identiﬁability of attention weights.
Further, while these works focus on RNN-based language models with a single layer of attention, we instead consider multi-head multi-layer self-attention models.
Our token classiﬁcation and token mixing experiments show that non-identiﬁable tokens increase with depth, further reinforcing the point that the factors that contribute to the mixing of information are complex and deserve further study.
Voita et al.
(2019) and Michel et al.
ﬁnd that only a small number of heads in BERT have a relevant effect on the output.
These results are akin to ours about the non-identiﬁability of attention weights, showing that a signiﬁcant part of attention weights do not affect downstream components.
One line of work investigates the internal representations of transformers by attaching probing classiﬁers to different parts of the model.
Tenney et al.
ﬁnd that BERT has learned to perform steps from the classical NLP pipeline.
Similarly, Jawahar et al.
(2019) show that lower layers of BERT learn syntactic features, while higher layers learn semantic features.
They also argue that long-range features are learned in later layers, which agrees with our attribution-based experiments.
C ONCLUSION
We used the notion of identiﬁability to gain a better understanding of transformers from different yet complementary angles.
We proved that attention weights are non-identiﬁable when the sequence length is longer than the attention head dimension. Thus, inﬁnitely many attention distributions can lead to the same internal representation and model output.
We propose effective attention as a tool to improve the interpretability of attention weights.
Second, we show that tokens remain largely identiﬁable by their angle through a learned linear transformation.
However, input tokens gradually lose their identity in the last layers.
Finally, we present Hidden Token Attribution, a gradient-based method to quantify information mixing. We use it to demonstrate that input tokens mix heavily inside transformers.
Hence, attention-based attribution, which suggests that a word at some layer is attending to another word, can be misleading.
Rather, a hidden mixture of embeddings attends to other hidden mixtures of embeddings.
We further show that context is progressively aggregated into the hidden embeddings while some identity information is preserved.
Moreover, we show that context aggregation is mostly local and that distant dependencies become relevant only in the last layers, which highlights the importance of local information for natural language understanding.
In summary, representations learned via self-attention are the result of complex factorizations that can be further characterized.
Our results suggest that some of the conclusions in prior work (Vaswani et al., 2017; Vig, 2019; Marecek & Rosa, 2018; Clark et al., 2019; Raganato & Tiedemann, 2018; Voita et al., 2019; Tang et al., 2018; Wangperawong, 2018; Padigela et al., 2019; Baan et al., 2019;
Lin et al., 2019; Dehghani et al., 2019; Zenkel et al., 2019; Klein & Nabi, 2019; Coenen et al., 2019) may be worth re-examining from this perspective.
A I DENTIFIABILITY OF S ELF -A TTENTION A. B ACKGROUND ON ATTENTION IDENTIFIABILITY
Often, the identiﬁability issue arises for a model with a large number of unknown parameters and limited observations.
Taking a simple linear model y = x 1 β 1 + x 2 β 2 as an example, when there is only one observation ( y, x 1 , x 2 ) , model parameters β 1 and β 2 cannot be uniquely determined.
Moreover, in the matrix form Y =
Xβ , by deﬁnition the parameter β is identiﬁable only if Y =
Xβ 1 and Y =
Xβ 2 imply β 1 = β 2 .
So if the null space contains only the zero solution { β | Xβ = 0 } = { 0 } , i.e., Xβ 1 − Xβ
= X ( β 1 − β 2 ) =
= ⇒ β 1 − β 2 = 0 , the model is identiﬁable.
Therefore, the identiﬁability of parameters in a linear model is linked to the dimension of the null space, which in the end is determined by the rank of X .
A. A DDITIONAL R ESULTS OF THE E FFECTIVE A TTENTION VS . R AW A TTENTION R ESULTS
In Figure 5 we provide a recreation of the Figure regarding the attention of tokens towards the [SEP] token found in (Clark et al., 2019) with average attention as well as average effective attention.
Again we see that most of the raw attention lies effectively in the null space, rendering the conclusions drawn questionable.
The Figures are produced using the code from Clark et al.
Layer
E ff ec ti v e A tt e n ti on other → [SEP] [SEP] → [SEP] (a)       0 .
Layer
R [Figure]
n ti on (b)
A. A CLOSER LOOK AT EFFECTIVE ATTENTION WEIGHTS
Here we discuss an example of how effective attention might lead to interpretive conclusions that differ from raw attention.
Figure 6 plots the (raw, effective, null) attention weights from one of the attention heads in BERT’s layer 4, for the following passage: ”
[CLS] research into military brats has consistently shown them to be better behaved than their civilian counterparts.
[SEP] hypotheses as to why brats are better behaved: ﬁrstly, military parents have a lower threshold for misbehavior in their children; secondly, the mobility of teenagers might make them less likely to attract attention to themselves, as many want to ﬁt in and are less secure with their surroundings; and thirdly, normative constraints are greater, with brats knowing that their behavior is under scrutiny and can affect the military member’s career.
teenage years are typically a period when people establish independence by taking some risks away from their [SEP]”.
For readability, on the y-axis, we focus on the single sentence ”the mobility of teenagers might make them less likely to attract attention to themselves, as many want to ﬁt in and are less secure with their surroundings”.
The following seems worth noticing:
• Raw attention weights are by and large concentrated either on the structural components, [CLS] and [SEP], or on the semi-monotonic, near diagonal alignments.
• Effective attention weights are more uniform, in general.
They are still concentrated near the diagonal elements, although less so than in raw attention.
However, the attention on [CLS] and [SEP] has vanished.
The collapse of the [CLS] and [SEP] weights might bring to the surface other potentially interesting items.
As an example, we point out the highest weight on the attention matrix, that is not on the diagonal.
This involves (highlighted by means of the yellow lines) the main verb of the selected sentence, ”make”, whose subject is ”them” (teenagers), and the pronoun ”them” (the direct object of the ﬁrst sentence, ”military brats”, 48 positions away).
The two are co-referential.
• Null attention weights are also more uniform than raw attention ones.
Interestingly, they seem to carry all the mass of the [CLS] and [SEP] tokens.
There is a visible degree of redundancy between the null attention weights and the effective ones, but also clear complementary elements.
One should not extrapolate too much from a single observation.
Further research is needed on this topic.
However, this example is a clear proof of concept that raw and effective attention can diverge qualitatively, in signiﬁcant ways.
It agrees with the hypothesis that the weights on the structural components may act as sinks, as observed in (Clark et al., 2019), but also tells us how this happens.
Furthermore, it also suggests that attention in the null space can obfuscate other valuable interactions that may be recoverable by inspecting effective attention.
U nd e r r e
v
i e w a s a c on f e r e n ce p a p e r a t
I C L
R [ C L S ] r e s e a r c h i n t o m ili t a r y b r a t ## s h a s c o n s i s t
e n t l y s h o w n t
h e m t
o b e b e tt e r b e h a v e d t h a n t h e i r
c i v ili a n c o un t e r p a r t s .
[ S E P ] ## o t h
## e
s e s a s t o
w h y b r a t ## s a r e b e tt e r b e h a v e d : f i r s t l y , m ili t a r y p a r e n t s h a v e a l o w e r t
h r e s h o l d f o r
m i s ## b e ## h a ## v i o
## r i n t h e i r c h il d r e n ; s e c o n d l y , t
h e m o b ili t y o f t ee n a g e r
s m i g h t m a k e t
h e m l e ss li k e l y t
o a tt r a c t a tt e n t i o n t o t
h
e
m s e l v e s , a s m a n y w a n tt o f i t i n a n d a r e l e ss s e c u r e w
i
t h t
h e i r s
u rr o un d i n g s ; a n d t h
i r d ## l y , n o r m a ## t
i v e c
o n s t r a i n t s a r e g r e a t e r , w i t h b r a t
## s k n o w i n g t h a t t
h e i r b e h
a v i o r
i s un d e r s c r u t i n y a n d c a n a ff e c t t h e m ili t
a r y m e m b e r ' s c a r ee r .
t ee n a g e y e a r s a r e t
y p
i c a ll y a p e r i o d w h e n p e
o p l e e s t a b li s h
i n d e p e n d e n c e b y t a k i n g s o m e r
i s k s a w a y f r o m t
h e i r [ S E P ] , the mobilityof teenagersmightmakethemlesslikelytoattractattentiontothemselves ,as manywanttofitinandarelesssecurewiththeir surroundings; 0.
[ C L S ] r e s
[Table]
a
[Table]
m ili
[Table]
a
[Table]
t #
[Table]
c o
[Table] m [Table]
e
[Table]
a v
[Table]
c
[Table]
t
[Table]
p
[Table]
[Table]
e
[Table]
e
[Table]
f
[Table]
m
[Table]
t i n a n d a r e l e ss s e c u r e w i t h t
h e i r s
u rr o un d i n g s ; a n d t h
i r d ## l y , n o r m a ## t
i v e c
o n s t r a i n t s a r e g r e a t e r , w i t h b r a t
## s k n o w i n g t h a t t
h e i r b e h
a v
i
o r
i s un d e r s c r u t i n y a n d c a n a ff e c t t h e m ili t
a r y m e m b e r ' s c a r ee r .
t ee n a g e y e a r s a r e t
y p
i c a ll y a p e r i o d w h e n p e
o p l e e s t a b li s h
i n d e p e n d e n c e b y t a k i n g s o m e r
i s k s a w a y f r o m t
h e i r [ S E P ] , the mobilityof teenagersmightmakethemlesslikelytoattractattentiontothemselves ,as manywanttofitinandarelesssecurewiththeir surroundings;
[Table]
[ C L S ] r e s e a r c h i n t o m ili t a r y b r a t ## s h a s c o n s i s t
e n t l y s h o w n t
h e m t
o b e b e tt e r b e h a v e d t h a n t h e i r
c i v ili a n c o un t e r p a r t s .
[ S E P ] ## o t h
## e
s e s a s t o
w h y b r a t ## s a r e b e tt e r b e h a v e d : f i r s t l y , m ili t a r y p a r e n t s h a v e a l o w e r t
h r e s h o l d f o r
m i s ## b e ## h a ## v i o
## r i n t h e i r c h il d r e n ; s e c o n d l y , t
h e m o b ili t y o f t ee n a g e r
s m i g h t m a k e t
h e m l e ss li k e l y t
o a tt r a c t a tt e n t i o n t o t
h
e
m s e l v e s , a s m a n y w a n tt o f i t i n a n d a r e l e ss s e c u r e w
i
t h t
h e i r s
u rr o un d i n g s ; a n d t h
i r d ## l y , n o r m a ## t
i v e c
o n s t r a i n t s a r e g r e a t e r , w i t h b r a t
## s k n o w i n g t h a t t
h e i r b e h
a v
i
o r
i s un d e r s c r u t i n y a n d c a n a ff e c t t h e m ili t
a r y m e m b e r ' s c a r ee r .
t ee n a g e y e a r s a r e t
y p
i c a ll y a p e r i o d w h e n p e
o p l e e s t a b li s h
i n d e p e n d e n c e b y t a k i n g s o m e r
i s k s a w a y f r o m t
h e i r [ S E P ]
[Figure]
B T OKEN I DENTIFIABILITY E XPERIMENTS B. E XPERIMENTAL S ETUP AND T RAINING D ETAILS
The linear perceptron and MLP are both trained by either minimizing the L2 or cosine distance loss using the ADAM optimizer (Kingma & Ba, 2015) with a learning rate of α = 0 . 0001 , β 1 = 0 .
9 and β 2 = 0 .
We use a batch size of 256.
We monitor performance on the validation set and stop training if there is no improvement for 20 epochs.
The input and output dimension of the models is d = 768 ; the dimension of the contextual word embeddings.
For both models we performed a learning rate search over the values α ∈ [0 . 003 , 0 . 001 , 0 . 0003 , 0 . 0001 , 0 . 00003 , 0 .
The weights are initialized with the Glorot Uniform initializer (Glorot & Bengio, 2010).
The MLP has one hidden layer with 1000 neurons and uses the gelu activation function (Hendrycks & Gimpel, 2016), following the feed-forward layers in BERT and GPT.
We chose a hidden layer size ofin order to avoid a bottleneck.
We experimented with using a larger hidden layer of sizeand adding dropout to more closely match the feed-forward layers in BERT. This only resulted in increased training times and we hence deferred from further architecture search.
We split the data by sentences into train/validation/test according to a 70/15/15 split.
This way of splitting the data ensures that the models have never seen the test sentences (i.e., contexts) during training.
In order to get a more robust estimate of performance we perform the experiments in Figure 2a using 10-fold cross validation.
The variance, due to the random assignment of sentences to train/validation/test sets, is small, and hence not shown.
B. G ENERALIZATION E RROR
Figure 7 shows the token identiﬁability rate for train and test set for both models, linear and MLP, when using L2 distance.
Both models are overﬁtting to the same degree.
The fact that the linear model has about the same generalization error as the MLP suggests that more training data would not signiﬁcantly increase performance on the test set.
Further, we trained the MLP on layerusing 50%, 80%, 90% and 100% of the training data set.
The MLP achieved the following token identiﬁability rate on the test set: 0.74, 0.8, 0.81, 0.82.
This indicates that the MLP would not proﬁt much from more data.
We do not report the generalization error for the models trained to minimize cosine distance, as the linear and non-linear perceptrons perform almost equally.
[Figure] [Figure]
B
[Figure]
A DDITIONAL RESULTS FOR F IGURE
[Figure]
Figure 2b in the main text only shows results of the linear perceptron trained to minimize cosine distance on layers l = [1 , 6 , 11 , 12] and tested on all other layers.
Figures 8, 9 and 10 show the corresponding results for the linear perceptron trained to minimize cosine distance, and for the
MLP trained to minimize L2 and cosine distance respectively. Overall, all ﬁgures show the same qualitative trends as presented in Section 4 of the main text: Generalizing to later layers works considerably worse than the other way around.
The linear perceptrons seem to generalize better across layers, likely due to the MLPs overﬁtting more to the particular layers they are trained on. [Figure] l
[Figure]
l= l= l=
[Figure] [Figure] [Figure] [Figure]
Layer
I d e n ti ﬁ a b ilit y
R a t e l= l= l= l=
[Figure]
Figure 11 sbows results for identifying tokens across single layers of BERT, i.e., the input to g is ( e 1 i , x i ) in the ﬁrst layer, and subsequently ( e li , e l − 1 i ) , where l = [1 , ..., 12] .
This experiment gives further insight into what kinds of transformations are applied by each transformer layer separately, as opposed to the cumulative transformations shown in Section 4 of the main text.
Interestingly, even the naive baselines perform well across single layers.
This shows that BERT only applies small changes to the contextual word embeddings, whereas overall the angle (as indicated by the naive baseline using cosine distance) is affected less than the magnitude of the word embeddings (indicated by the naive baseline using L2 distance).
Figure 11 shows that tokens are on average more difﬁcult to identify across later layers. In the main text we hypothesize that the qualitative change seen in later layers could be due to a task-speciﬁc parameter adaptation during the second (next sentence prediction) pre-training phase. A possible reason is that during this pre-training-phase, BERT only needs the [CLS] token in the last layer, which is qualitatively very different form the ﬁrst (masked language modeling) pre-training phase, where potentially all the tokens are needed in the last layer. To further verify this hypothesis we experimented with BERT ﬁne-tuned on two datasets, MRPC and
CoLA (Warstadt et al., 2018).
During the ﬁne-tuning phase, similar to the next sentence prediction pre-training phase, only the [CLS] token is needed at the last layer.
If task-dependent parameter adaptation indeed has a different inﬂuence on the last layer(s) than on earlier layers, then we should be able to see a difference between the ﬁnetuned and non-ﬁnetuned cases.
Figures 12 andcompare the naive baselines across single layers for BERT ﬁnetuned on MRPC and CoLA, respectively.
Indeed, one can see a remarkable decrease in identiﬁability across the last layer for L2-based nearest neighbour lookup, further indicating that the last layers are indeed more strongly affected by different ﬁne-tuning objectives.
Nearest neighbour lookup based on cosine distance is affected much less, indicating that in terms of token identiﬁability, the last layers are only slightly affected by ﬁne-tuning.
[Figure]
Layer
I d e n ti ﬁ a b ilit y
R a t e
[Equation]
[Figure]
[Figure]
[Figure]
[Figure] [Figure] B. T OKEN I DENTITY - R ECOVER N EIGHBOURING I NPUT T OKENS In Section 4 of the main text we show that tokens at position i remain largely identiﬁable throughout the layers of BERT.
In this section we show results of a related experiment, where we test how much information about tokens at neighbouring positions is contained in a contextual word embedding.
More formally, the input to g is ( e li , x i ± k ) , where k ∈ { 1 , 2 , 3 } .
Thus, we try to recover input token
x i + k from hidden token e li .
Figures 14, 15, 16 and 17 show the results of for ˆ g lincos,l , ˆ g MLPcos,l , ˆ g MLPL 2 ,l and ˆ g linL 2 ,l , respectively.
In the ﬁgures, blue corresponds to “previous” tokens and red to “next” tokens.
From the ﬁgures we can see that tokens do contain information about neighbouring tokens that lets us recover the neighbouring tokens based on a transformation and subsequent nearest neighbour lookup.
The identiﬁability rate drops both with increasing k , but also with increasing depth.
Interestingly, recovering left (red) and right (red) neighbours shows different behaviour, indicating that BERT is treating left and right context differently, despite having been pre-trained using a bidirectional language modeling objective.
Overall, neighbouring tokens can be recovered to a much lower degree than same-position tokens (cf. Section4).
[Figure]
Layer
[Figure]
ii-
ii+ i+ i+
[Figure]
Layer
[Figure]
ti ﬁ a b ilit y
R a t e ii- ii+ i+ i+
[Figure]
[Figure]
Layer
I d e n ti ﬁ a b ilit y
R a t e ii- ii+ i+ i+
[Figure]
[Figure]
Layer
I d e n ti ﬁ a b il it y
R a t e ii- ii+ i+ i+
C C ONTEXT C ONTRIBUTION A NALYSIS
C.
H IDDEN T OKEN A TTRIBUTION :
D
ETAILS
The attribution method proposed in Section 5.1 to calculate the contribution of input tokens to a given embedding does not look at the output of the model but at the intermediate hidden representations and therefore is task independent.
Since the contribution values do not depend on the task that is evaluated, we can compare these values directly to attention distributions, which are also taskindependent.
In this way, we can compare to other works in the literature (Vig, 2019; Clark et al., 2019; Klein & Nabi, 2019; Coenen et al., 2019; Lin et al., 2019) by using the publicly available pretrained BERT model in our analyses without ﬁne-tuning it to a speciﬁc task.
Furthermore, since we are not interested in analysing how the input affects the output of the model but in quantifying the absolute contribution of the input tokens to the hidden embeddings, we use the
L 2 norm of the gradients.
If we were analyzing whether the input contributed positively or negatively to a given decision, the dot-product of the input token embedding with the gradient would be the natural attribution choice (P¨orner et al., 2018).
C. C ONTEXT I DENTIFIABILITY : D ETAILS
To calculate the relative contribution values shown in Figure 4a we ﬁrstly calculate the mean of the left and right neighbours for each of the groups of neighbours, i.e., ﬁrst, second, third, fourth and ﬁfth, sixth to 10th and, from 11th onwards.
Then we aggregate the values averaging over all the tokens in the MRPC evaluation set.
Finally, we normalize for each group so that the sum of the contribution values of each group is one.
In this way, we can observe in which layer the contribution of a given group of neighbours is the largest.
Our results on context identiﬁability from Section 5.3 complement some of the studies in previous literature.
In (Jawahar et al., 2019) the authors observe that transformers learn local syntactic tasks in the ﬁrst layers and long range semantic tasks in the last layers.
We explain this behavior from the point of view of context aggregation by showing that distant context acquires more importance in the last layers (semantic tasks) while the ﬁrst layers aggregate local context (syntactic tasks).
Furthermore, the results showing that the context aggregation is mainly local, specially in the ﬁrst layers, provide an explanation for the increase in performance observed in (Yang et al., 2018).
In that work, the authors enforce a locality constraint in the ﬁrst layers of transformers, which pushes the model towards the local operators that it naturally tends to learn, as we show in Figure 4b, improving in this way the overall performance.
C. C ONTEXT C ONTRIBUTION TO CLS TOKEN
In this section we use Hidden Token Attribution to look at the contribution of the context to the [CLS] token, which is added to the beginning of the input sequence by the BERT pre-processing pipeline.
This is an especially interesting token to look at because the decision of BERT for a classiﬁcation task is based on the output in the [CLS] token.
Furthermore, like the [SEP] token, it does not correspond to a natural language word and its position in the input sequence does not have any meaning.
Therefore, the conclusion that context is on average predominantly local (cf.
Section 5.3), is likely to not hold for [CLS].
The second and ﬁnal pre-training task that BERT is trained on is next sentence prediction.
During this task, BERT receives two sentences separated by a [SEP] token as input, and then has to predict whether the second sentence follows the ﬁrst sentence or not.
Therefore, it is expected that the context aggregated into the [CLS] token comes mainly from the tokens around the ﬁrst [SEP] token, which marks the border between the ﬁrst and second sentence in the input sequence.
In Figure 18 we show the contribution to the [CLS] token from all of its neighbours averaged over all the examples in the MRPC evaluation set for the ﬁrst, middle and last layers.
In Figure 18a, the [CLS] token is placed at position
and we see how the context contribution comes mainly from the tokens around position 30, which is roughly the middle of the input examples.
In Figure 18b we center the contribution around the ﬁrst [SEP] token and indeed, it becomes clear that the [CLS] token is aggregating most of its context from the tokens around [SEP], i.e., from the junction between both sentences.
In particular, the two tokens with the highest contribution are the tokens directly before and after [SEP].
Also, it seems that the second sentence contributes more to [CLS] than the ﬁrst one.
These results give an insight on what information BERT uses to solve next sentence prediction and serves as an illustrative example of how Hidden Token Attribution can be used to analyze transformers.
Neighbour
T o t a l c on t r i bu ti on t o [
C L S ] l =l =l =(a) - - [SEP]   0 .
Neighbour
T o t a l c on t r i bu ti on t o [
C L S ] l =l =l =(
[Figure]
C. T RACKING C ONTEXT C ONTRIBUTION
Here we show examples of how Hidden Token Attribution can track how context is aggregated for a given word at each layer.
For reasons of space we show only few words of a randomly picked sentence of the MRPC evaluation set, which is tokenized as follows:
[CLS] he said the foods
##er ##vic ##e
pie business doesn ’ t fit the company ’ s long - term growth strategy .
[SEP] " the foods ##er ##vic ##e pie business does not fit our long - term growth strategy .
[SEP] [ C L S ] h e s a i d t h e f oo d s ## e r ## v i c
## e p i e b u s i n e ss d
o e s n ' t f i t
t h e c o m p a n y ' s l o n g - t e r m g
r o w t h s t r a t e g y .
[ S E P ] " t
h e f oo d s ## e r ## v i c
## e p i e b u s i n e ss d o e s n o t f
i t o u r l o n g - t e r m g r o w t h s t r a t e g y .
[ S E P ]
[Figure]
[Figure]
[Figure]
[ C L S ] h e s a i d t
h e f oo d s ## e r ## v i c
## e p i e b u s i n e ss d
o e s n ' t f i t
t h e c o m p a n y ' s l o n g - t e r m g
r o w t h s t r a t e g y .
[ S E P ] " t
h e f oo d s ## e r ## v i c
## e p i e b u s i n e ss d o e s n o t f
i t o u r l o n g - t e r m g r o w t h s t r a t e
[Figure]
[ C L S ] h e s a i d t
h e f oo d s ## e r ## v i c
## e p i e b u s i n e ss d
o e s n ' t f i t
t h e c o m p a n y ' s l o n g - t e r m g
r o w t h s t r a t e g y .
[ S E P ] " t
h e f oo d s ## e r ## v i c
## e p i e b u s i n e ss d o e s n o t f
i t o u r l o n g - t e r m g r o w t h s t r a t e g y .
[ S E P ]
[Figure] [Figure]
e b u s i n e ss d o e s n ' t f i t t h e c o m p a n y ' s l o n g - t e r m g r o w t h s t r a t e g y .
[ S E P ] " t
h e f oo d s ## e r ## v i c
## e p i e b u s i n e ss d o e s n o t f
i t o u r l o n g - t e r m g r o w t h s t r a t e g y .
[ S E P ]
[Figure] [ C L S ] h e s a i d t h e f oo d s ## e r ## v i c ## e p i e b u s i n e ss d
o e s n ' t f i t
t h e c o m p a n y ' s l o n g - t e r m g
r o w t h s t r a t e g y .
[ S E P ] " t
h e f oo d s ## e r ## v i c
[Figure]
p i
[Figure]
C. T OKEN C ONTRIBUTIONS BY POS T AG
Here we show the contribution of input tokens to hidden representations in all layers splitted by partof-speech (POS) tag (Toutanova et al., 2003).
The POS tags are ordered according to the contribution in layer 12. [Figure] [Figure] [Figure]
[Figure]
[Figure]
[Figure] [Figure] [Figure] [Figure]
C o n t r i b u t i o n
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
D G ENERALIZATION TO OTHER D ATASETS
In this appendix we reproduce several experiments from the main text using the development sets of two additional datasets from the GLUE benchmark: The Corpus of Linguistic Acceptability (CoLA) (Warstadt et al., 2018), and the matched Multi-Genre Natural Language Inference corpus (MNLI-matched) (Williams et al., 2018).
CoLA is a dataset about grammatical acceptability of sentences and MNLI consists of pairs of sentences where the second sentence entails, contradicts or is neutral about the ﬁrst one.
These datasets differ signiﬁcantly from MRPC.
The development set of CoLa has 1043 examples with sequence length
d s between 5 and 35 tokens, and 11 tokens on average.
The development set of MNLI-m consits of 9815 examples although we restrict the experiments to the ﬁrst 4000 examples without loss of generality.
These contain a total oftokens, with a sequence length comprised between 6 and 128 tokens and an average of 39 tokens per example.
The results presented in this appendix are qualitatively similar to those presented in the main text, which shows that our empirical conclusions about BERT are general across data domains.
D. T OKEN I DENTIFIABILITY
Here we reproduce the main token identiﬁability results of Section 4 on two additional datsets:
CoLA and MNLI.
Qualitatively, the results are in line with those for MRPC.
Note that a random classiﬁer would achieve an accuracy of 1 / ¯ d s , where ¯ d s denotes the average sentence length.
Thus, the random guessing baselines for MRPC, CoLA and MNLI are 1 .
7% , 9% and 2 .
6% respectively.
[Equation]
Figure 38 shows the token identiﬁability results for CoLA.
Layer
I d e n ti ﬁ a b ilit y
R a t e ˆ g MLPcos
[Equation]
l ˆ g lincos,l ˆ g naivecos,l ˆ g MLPL 2 ,l ˆ g linL 2 ,l ˆ g naiveL
[Figure]
Figure 39 shows the token identiﬁability results for the ﬁrst 500 sentences (19,839 tokens) of MNLImatched.
[Figure]
Layer
[Figure]
D. A TTRIBUTION A NALYSIS D.2. C O LA E XPERIMENTS
Figure D.2.1 shows the token mixing analysis for the CoLA dataset.
The behavior is very similar to
MRPC with the only difference that both, the contribution of the original token and the percentage of tokens that are not maximum contributors to their embeddings are slightly larger across layers.
However, this increase is explained by CoLA consisting of much shorter sequences on average than
MRPC.
Layer
C on t r i bu ti on [ % ]
[Figure]
[Figure]
Layer
[Figure]
Figure D.2.1 presents the context aggregation for the CoLA development set.
We observe the same general trend as for MRPC, with the context being aggregated mostly locally and long range dependencies increasing in the later layers.
The fact that examples in CoLA have an average sequence length of 11 tokens explains the smaller relative contribution of tokens beyond the 10th neighbour.
[Figure]
Layer
R e l .
C on t r i bu ti on ( % )
[Figure] Neighbour [Figure]
[Figure]
As shown by Figures D.2.2 and D.2.2, the results with the MNLI matched dataset are very similar to the ones presented in the main text. No meaningful discrepancy exists in this case.
[Figure] [Figure]
C on t r i bu ti on [ % ]
[Figure] [Figure] [Figure]
R e l .
C on t r i bu ti on ( % )
[Figure]
Neighbour
[Figure] [Figure]