Neurons in Large Language Models: Dead, N-gram, Positional (2024)

Abstract: We analyze a family of large language models in such a lightweight manner that can be done on a single GPU. Specifically, we focus on the OPT family of models ranging from 125m to 66b parameters and rely only on whether an FFN neuron is activated or not. First, we find that the early part of the network is sparse and represents many discrete features. Here, many neurons (more than in some layers of the 66b model) are “dead”, i.e. they never activate on a large collection of diverse data. At the same time, many of the alive neurons are reserved for discrete features and act as token and n-gram detectors. Interestingly, their corresponding FFN updates not only promote next token candidates as could be expected, but also explicitly focus on removing the information about triggering them tokens, i.e., current input. To the best of our knowledge, this is the first example of mechanisms specialized at removing (rather than adding) information from the residual stream. With scale, models become more sparse in a sense that they have more dead neurons and token detectors. Finally, some neurons are positional: them being activated or not depends largely (or solely) on position and less so (or not at all) on textual data. We find that smaller models have sets of neurons acting as position range indicators while larger models operate in a less explicit manner.

## introduction the range of capabilities of language models expands with scale and at larger scales models become so strong and versatile that a single model can be integrated into various applications and decisionmaking processes @xcite @xcite @xcite . this increases interest and importance of understanding the internal * work done as part of internship at meta ai. workings of these large language models (llms) and, specifically, their evolution with scale. unfortunately, scaling also increases the entry threshold for interpretability researchers since dealing with large models requires a lot of computational resources. in this work, we analyze a family of opt models up to 66b parameters and deliberately keep our analysis very lightweight so that it could be done using a single gpu. we focus on neurons inside ffns, i.e. individual activations in the representation between the two linear layers of the transformer feedforward blocks (ffns). differently from e.g. neurons in the residual stream, ffn neurons are more likely to represent meaningful features: the elementwise nonlinearity breaks the rotational invariance of this representation and encourages features to align with the basis dimensions @xcite . when such a neuron is activated, it updates the residual stream by pulling out the corresponding row of the second ffn layer; when it is not activated, it does not update the residual stream (figure 6 ). 1 therefore, we can interpret functions of these ffn neurons in two ways: (i) by understanding when they are activated, and (ii) by interpreting the corresponding updates coming to the residual stream. first, we find that in the first half of the network, many neurons are "dead", i.e. they never activate on a large collection of diverse data. larger models are more sparse in this sense: for example, in the 66b model more that 70% of the neurons in some layers are dead. at the same time, many of the alive neurons in this early network part are reserved for discrete features and act as indicator functions for tokens and n-grams: they activate if and only if the input is a certain token or an n-gram. the role of the updates coming from these token detectors to the residual stream is also surprising: at the same time as they promote next token candidates (which is to be expected according to @xcite ), they are explicitly targeted at removing information about current input, i.e. their triggers. this means that in the bottom-up processing where a representation of the current input token gets gradually transformed into a representation for the next token, current token identity is removed by the model explicitly (rather than ends up implicitly "buried" as a result of additive updates useful for the next token). to the best of our knowledge, this is the first example of mechanisms specialized at removing (rather than adding) information from the residual stream. finally, we find that some neurons are responsible for encoding positional information regardless of textual patterns. similarly to token and n-gram detectors, many of these neurons act as indicator functions of position ranges, i.e. activate for positions within certain ranges and do not activate otherwise. interestingly, these neurons often collaborate. for example, the second layer of the 125m model has 10 positional neurons whose indicated positional ranges are in agreement: together, they efficiently cover all possible positions and no neuron is redundant. in a broader picture, positional neurons question the key-value memory view of the ffn layers stating that "each key correlates with textual patterns in the training data and each value induces a distribution over the output vocabulary" @xcite . neurons that rely on position regardless of textual pattern indicate that ffn layers can be used by the model in ways that do not fit the key-value memory view. overall, we argue that the roles played by these layers are still poorly understood. to sum up, we find neurons that: larger models have more dead neurons and token detectors and are less focused on absolute position. ## data and setting models. we use opt @xcite , a suite of decoder-only pre-trained transformers that are publicly available. we use model sizes ranging from 125m to 66b parameters and take model weights from the huggingface model hub. 2 data. we use data from diverse sources containing development splits of the datasets used in opt training as well as several additional datasets. overall, we used (i) subsets of the validation and test part of the pile @xcite including wikipedia, dm mathematics, hackernews, (ii) reddit @xcite , (iii) code data from codeparrot foot_2 . for the experiments in section 3 when talking about dead neurons, we use several times more data. specifically, we add more data from wikipedia, dm mathematics and codeparrot, as well as new domains from the pile foot_3 : europarl, freelaw, pubmed abstracts, stackexchange. overall, the data used in section 3 has over 20m tokens, in the rest of the paper -over 5m tokens. single-gpu processing. we use only sets of neuron values for some data, i.e. we run only forward passes of the full model or its several first layers. since large models do not fit in a single gpu, we load one layer at a time keeping the rest of the layers on cpu. this allows us to record neuron activations for large models: all the main experiments in this paper were done on a single gpu. ## dead neurons let us start from simple statistics such as neuron activation frequency (figure 1 ). many neurons are "dead". first, we find that many neurons never activate on our diverse data, i.e. they can be seen as "dead". figure 1a shows that the proportion of dead neurons is very substantial: e.g., for the 66b model, the proportion of dead neurons in some layers is above 70%. we also see that larger models are more sparse because (i) they have more dead neurons and (ii) the ones that are alive activate less frequently (figure 1b ). only first half of the model is sparse. next, we notice that this kind of sparsity is specific only to early layers. this leads to a clear distinction between the first and the second halves of the network: while the first half contains a solid proportion of dead neurons, the second half is fully "alive". additionally, layers with most dead neurons are the ones where alive neurons activate most rarely. packing concepts into neurons. this difference in sparsity across layers might be explained by "concept-to-neuron" ratio being much smaller in the early layers than in the higher layers. intuitively, the model has to represent sets of encoded in a layer concepts by "spreading" them across available neurons. in the early layers, encoded concepts are largely shallow and are likely to be discrete (e.g., lexical) while at the higher layers, networks learn high-level semantics and reasoning @xcite @xcite @xcite . since the number of possible shallow patterns is not large and, potentially, enumerable, in the early layers the model can (and, as we will see later, does) assign dedicated neurons to some features. the more neurons are available to the model, the easier it is to do so -this agrees with the results in figure 1 . differently, the space of fine-grained semantic concepts is too large compared to the number of available neurons which makes it hard to reserve many dedicated neuron-concept pairs. foot_4 are dead neurons completely dead? note that the results in figure 1a can mean one of the two things: (i) these neurons can never be activated (i.e. they are "completely dead") or (ii) they correspond to patterns so rare that we never encountered them in our large diverse collection of data. while the latter is possible, note that this does not change the above discussion about sparsity and types of encoded concepts. on the contrary: it further supports the hypothesis that models assign dedicated neurons to specific concepts. ## n-gram-detecting neurons now, let us look more closely into the patterns encoded in the lower model halves and try to understand the nature of the observed above sparsity. specifically, we analyze how neuron activations depend on an input n-gram. for each input text with tokens x 1 , x 2 , ..., x s , we record neuron activations at each position and if a neuron is activated (i.e., non-zero) at position k, we say that the n-gram (x k-n+1 , . . . , x k ) triggered this neuron. in sections 4.1-4.4 we talk about unigrams (i.e., tokens) and come to larger n-grams in section 4.5. ## positional neurons when analyzing dead neurons (section 3), we also noticed some neurons that, consistently across diverse data, never activate except for a few first token positions. this motivates us to look further into how position is encoded in the model and, specifically, whether some neurons are responsible for encoding positional information. ## the 350m model: the odd one out as we already mentioned above, the 350m model does not follow the same pattern as the rest of the models. specifically, it does not have dead neurons (section 3) and its neuron activations do not seem to be sparse with respect to triggering them n-grams as we saw for all the other models in figure 2 . 10 modeling bits affect interpretability. this becomes less surprizing when noticing that the 350m model is implemented differently from all the rest: it applies layernorm after attention and feedforward blocks, while all the other models -before. 11 apparently, such seemingly minor implementation details can affect interpretability of model components rather significantly. indeed, previous work also tried choosing certain modeling aspects to encourage interpretability. examples of such work include choosing an activation function to increase the number of interpretable neurons (elhage et al., 2022), large body of work on sparse softmax variants to make output distributions or attention more interpretable (martins and astudillo (2016); niculae and blondel (2017); peters et al. (2019); correia et al. (2019); martins et al. (2020), among others), or more extreme approaches with explicit modular structure that is aimed to be interpretable by construction (andreas et al. (2016); hu et al. (2018); kirsch et al. (2018); khot et al. (2021), to name a few). intuitively, choosing relu activation function as done in the opt models can be seen as having the same motivation as developing sparse softmax variants: exact zeros in the model are inherently interpretable. ## additional related work historically, neurons have been a basic unit of analysis. early works started from convolutional networks first for images @xcite and later for text classification @xcite . similar to our work, @xcite also find n-gram detectors; although, for small convolutional text classifiers this is an almost trivial observation compared to transformer-based llms. for recurrent networks, interpretable neurons include simple patterns such as line lengths, brackets and quotes @xcite , sentiment neuron @xcite , and others @xcite . for bert, @xcite find that some ffn neurons store factual knowledge. larger units of analysis include attention heads ( voita et al. (2018, 2019b 2019 ), etc), feed-forward layers @xcite and circuits responsible for certain tasks @xcite @xcite . ## implications and conclusions overall, neurons in llms can (i) be dead (neveractivating), (ii) act as token-and n-gram detectors, (iii) encode position regardless of textual content. note that differently from most of the previous influential mechanistic interpretability work, we experiment with an entire family of models (instead of a single model as in @xcite ; @xcite among others) and consider very large models up to 66b. we also provide a way to analyze large models with very limited resourceswe believe this is of high value to academic community. finally, our main findings are not only about the presence of certain neurons in the opt models but also about high-level conclusions regarding current beliefs in the community. specifically, (1) information can be explicitly removed (rather than added) from the residual stream, (2) positional neurons question the key-value memory view of ffns, (3) we explain how lms trained without positional information still encode position, (4) we show that minor architecture changes can significantly influence interpretability, among others. on top of that, our analysis can easily be extended to other models: e.g., for other models later work validated our findings regarding positional neurons and suppressed concepts @xcite .