Spectral Filters, Dark Signals, and Attention Sinks (2024)

Abstract: Projecting intermediate representations onto the vocabulary is an increasingly popular interpretation tool for transformer-based LLMs, also known as the logit lens (Nostalgebraist). We propose a quantitative extension to this approach and define spectral filters on intermediate representations based on partitioning the singular vectors of the vocabulary embedding and unembedding matrices into bands. We find that the signals exchanged in the tail end of the spectrum, i.e. corresponding to the singular vectors with smallest singular values, are responsible for attention sinking (Xiao et al., 2023), of which we provide an explanation. We find that the negative log-likelihood of pretrained models can be kept low despite suppressing sizeable parts of the embedding spectrum in a layer-dependent way, as long as attention sinking is preserved. Finally, we discover that the representation of tokens that draw attention from many tokens have large projections on the tail end of the spectrum, and likely act as additional attention sinks.

## introduction large foundation models dominate the state of the art in numerous ai tasks. while we understand how these models work in terms of elementary operations, and black-box evaluations help characterize observable behaviours, we lack a clear understanding of the connection between the two. there is a growing body of work providing insights into properties of model components, e.g. @xcite @xcite @xcite , as well as identifying and explaining fundamental phenomena, often with the support of simple models @xcite @xcite @xcite . most recent works assign a central role to the model's residual stream (rs) as the shared communication channel between model components. in this perspective, the probability distribution of a token is initialised from the projection of the embedding of the previous token through the unembedding matrix, and receives additive updates from attention heads and mlp components, each reading from the residual stream of the same or previous tokens. the role played by components is interpreted projecting their contribution on the probability distribution over vocabulary items, in what is referred to as the logit lens (nostalgebraist; @xcite . we extend this approach and introduce logit spectroscopy, the spectral analysis of the content of the residual stream and of the parameter matrices interacting with it. equipped with this tool, we look at the part of the residual stream spectrum that is most likely to be neglected by the logit lens: the linear subspace spanned by the right singular vectors of the unembedding matrix with the smallest singular values. drawing an analogy with "dark matter" in astrophysics, that interacts with light only indirectly, we dub projections onto this subspace dark parameters, features, activations etc. we were motivated by the thought that llms could learn to use signals in the dark linear subspace to maintain global features responsible for long-range dependencies while minimizing their interference with the next token prediction. we discovered instead that dark signals are instrumental to implementing the recently described phenomenon of attention sinks @xcite , of which we provide a detailed account. we also show that the negative log likelihood of pretrained models can be kept low despite suppressing large swaths of the unembedding spectrum, as long as the dark signals required for attention sinking are untouched. finally, we find a significant positive correlation between the average attention received by a token and the relative prevalence of dark signals in its residual stream, and find evidence that tokens uniformly receiving substantial attention are likely acting as additional attention sinks. ## the llamamodels we chose llama2 models @xcite as the object of our study as they were the most competitive models with open-access weights at the time we started this work. in particular we studied pretrained models (without instruction fine-tuning) with 7b, 13b, and 70b parameters. we point out a few differences between llama2 models and the original transformer architecture @xcite . rmsnorm @xcite is applied before every attention component, mlp component, and the final unembedding projection. rmsnorm includes a learned rescaling vector that is applied after the normalization proper. in all cases, we absorb this rescaling vector into the matrices downstream from the rescaling: this is mathematically equivalent and simplifies the analysis. llama2 models use swiglu activation functions @xcite . this means that mlp components have three parameter matrices instead of the more familiar two. finally, llama2 models use rotary positional embeddings @xcite . while important for the functioning of the model, their effect is independent of the phenomena we are focusing on. ## model parameter properties we use the adjective u-dark (e-dark) to characterize anything that happens in the linear subspace spanned by the 5% right singular vectors (rsvs) of w u (w e ) with the smallest svs, the dark basis. we gain an initial insight into whether dark signals are exchanged by projecting rows and columns of parameter matrices on the dark bases. w k , w q , and w v project the residual stream onto either the latent space used to compute attention scores or the latent space used to compute the attention output, so they "read" from the residual stream; similarly w 1 and w 3 map from the residual stream onto the activation layer of the mlp components. we project the columns of these matrices on the rsvs of w u to estimate and visualize their aptitude to read from the dark subspace 2 . conversely, w o and w 2 map from latent spaces back into the residual stream, therefore we project their rows to check their aptitude to write into the dark subspace. we computed and plotted the norms of the d vectors of dimension d h or d m obtained with these projections, e.g.: we discovered a great variety in where, in the bases formed by the rsvs of w u and w e , model components are equipped to read from and write into. figure 4 gives a sense of such variety. the projections of mlp matrices also clearly indicate that some mlp components can write mostly into the dark subspace (see fig. 5 for the projection of 13b/l0/w 2 on w u ). ## spectral filtering one possible explanation for the existence of components that communicate through the dark subspace is that they are useless, and the model learned to divert their output to subspaces with little bearing on the vocabulary logits. to see if this is the case, we perform a series of experiments, where we patch some of the intermediate representations by projecting them onto the rsvs of w u and w e with largest singular vectors, therefore removing dark signals. we measure the average negative loglikelihood (nll) of tokens in a sample of prompts: if dark signals are irrelevant noise then we expect nll to be largely unaffected when they are filtered out. we split the singular vectors of w u into 20 bands: {v u,1 , . . . , v u,20 }, of cardinality d/20, and similarly for w e . v u,1 contains the 5% of singular vectors with largest singular values, v u,2 the next 5%, and so on. let v u,j:k be the matrix obtained concatenating [v u,j . . . v u,k ], and (fig. 1 ) let φ u,j:@xmath0,j:k v ⊤ u,j:k (similarly for w e ). we can then project any d-dimensional vector onto the u-dark (e-dark) space by multiplying them by φ u,20:20 (resp. φ e,20:20 ). we form a hierarchy of nested filters φ u,1:k : multiplying a vector by φ u,1:k projects it onto the subspace of the kd/20 "least dark" dimensions. we also define filters that combine singular vectors of w u and w e (fig. 6 ): ## sink-preserving spectral filters the finding that dark signals are essential to enable attention sinks leads to the question: what would be the impact of spectral filters that preserve dark signals but filter away one or more bands before them? let ω u,k be a new family of spectral filters (fig. 12 ): ## dark signals and attention bars inspecting attention matrices in our sample we see that there are frequently a few tokens that also receive a large amount of attention, giving rise to characteristic attention bars (e.g. fig. 14 ). we hypothesized that such bars could correspond to tokens whose residual streams are similar to that of ψ(90%): the ea, a gs, the e, a, for a total, the gs, the e, the s, the e, the s, the 15, the 16, the 15, the 16, the 15, the 17, the rnd (90%): what is the most beautiful thing that has ever been? what is the most beautiful thing that has ever been? it is not about the most beautiful thing that has ever been. the most beautiful thing that has ever been. it is not about the most beautiful thing that has ever been. it is not token 0, and therefore become additional attention sinks. we define high-mean low-variance (hmlv) token-layer pairs those with mean received attention from subsequent tokens above τ µ = .018 and variance below τ σ = .01. we selected thresholds manually so that most hmlv tokens appear as "attention bars" in attention matrices 5 . this heuristic yields 12,236 hmlv token-layer pairs for llama2 13b on ccnet-405 out of a total of 404,748 tokenlayer pairs for the same tokens and layers considered. we ignore the bos token, the last 4 tokens of each sequence, and layers 0-3, since the attention sink is not in place in the 13b model yet. we define the u-dark ratio as a measure of the prevalence of u-dark signals in a representation: we also define the key dot-product as the average dot product between the keys projected through matrices w k : intuitively, this value quantifies the extent to which a residual stream appears similar to the one of token zero, as far as attention allocation is concerned. we measured the average u-dark ratio and key dot product for hmlvs, and contrasted them with the same statistics taken over all token-layer pairs. results are in fig. 15 . starting around l13 the u-dark ratio of hmlvs grows considerably, whereas it remains constantly below .75 for the overall population. at the same time, the gap in key dot product also grows, as the kdp for the overall population steadily decreases, while that for hmlvs stabilizes. analogous plots for the llama2 7b and 70b models show different profiles, but an even more marked difference between hmlvs and the general population (appendix f). these measures support the hypotheses that hmlvs are token-layer pairs that appear very similar to token 0, as far as attention allocation is concerned, and therefore become additional attention sinks. ## related work in this work we adopt the framework introduced in @xcite , which highlights the central role played by the residual stream as working memory and communication channel across components, and suggests the presence of subspaces with differentiated functions. the logit lens was first introduced by (nostalgebraist), and has been since used and extended in a number of interpretation studies, including @xcite @xcite @xcite . of these, @xcite analyse models projecting parameter matrices onto the vocabulary space, an approach we are also following in section 3. our hypothesis that the subspace most orthogonal to the vocabulary representation could encode non-sparse latent features that should not interfere with the distribution over the immediate next token appeared supported by @xcite , where they show that, in basic models, important nonsparse features tend to be assigned principal components rather than being in superposition with sparse features. @xcite showed that sparse autoencoders can recover a ground-truth assignment of features to linear subspaces by means of sparse autoencoders in an artificial and simplified setting and a 31m parameter llm, with @xmath1; @xcite apply a similar method to larger pythia 70m and pythia 410m models. our results with ω filters with the much larger llama2 models indicate that the residual stream could be used at less than full capacity, at least for some of the layers, raising the question of what model of superposition would best apply to them. the concept of attention sinks was introduced by @xcite , who noticed that performance of streaming models dropped abruptly once the first tokens moved out of the attention sliding window context. we take here the next step in explaining how this process works, showing how it happens thanks to a very limited and specific portion of the spectrum. @xcite show that it is possible to preserve and even improve the performance of the gpt-j (wang and komatsuzaki, 2021) model while dropping large portions of component matrices based on the svd decomposition of the matrices themselves. we also focus on spectral decomposition, but using the spectrum of the unembedding matrix, in accordance with the logit lens approach and the intuition that it represents a common ground for all the components in a model. ## conclusions and future work a better understanding of the inner workings of transformer models is important to find strategies to make them safer. in this work we explored a novel way to look at transformers, extending the logit lens approach into logit spectroscopy by introducing spectral filters. we explored the hypothesis that dark signals could be used to maintain global features while minimizing their interference with the next token, but we discovered that the main role of dark signals is in enabling attention sinking. we reconstructed how attention sinking works in llama2 models, and we showed that, as long as attention sinking is preserved, they still achieve low negative log-likelihood even when significant portions of the unembedding spectrum are suppressed. finally, we found a positive correlation between the attention received by a token and the relative prevalence of dark signals in its residual stream, especially in the upper layers. the results on spectral filtering with dark signal preservation, combined with the observation that transformers are invariant to basis changes in the residual stream, suggest that it could be possible to first "canonicalize" a model so that its residual stream dimensions match the v u columns, and then compress away from components dimensions that are low in the spectrum but are not used for attention sinking. we consider such a form of spectral compression an interesting direction for future work. finally, while our exploration of the connection between "attention bars", prevalence of dark signals, and similarity with the token 0 residual stream suggests that attention bars are additional attention sinks, increasing the granularity of the spectral bands in the dark subspace could lead to additional discoveries.