Outlier Dimensions Encode Task Specific Knowledge (2023)

Abstract: Representations from large language models (LLMs) are known to be dominated by a small subset of dimensions with exceedingly high variance. Previous works have argued that although ablating these outlier dimensions in LLM representations hurts downstream performance, outlier dimensions are detrimental to the representational quality of embeddings. In this study, we investigate how fine-tuning impacts outlier dimensions and show that 1) outlier dimensions that occur in pre-training persist in fine-tuned models and 2) a single outlier dimension can complete downstream tasks with a minimal error rate. Our results suggest that outlier dimensions can encode crucial task-specific knowledge and that the value of a representation in a single outlier dimension drives downstream model decisions.

## introduction large language models (llms) are highly overparameterized with llm representations utilizing only a small portion of the available embedding space uniformly @xcite @xcite . representations of transformer-based llms are dominated by a few outlier dimensions whose variance and magnitude are significantly larger than the rest of the model's representations @xcite . previous studies devoted to the formation of outlier dimensions in pre-trained llms suggest that imbalanced token frequency causes an uneven distribution of variance in model representations @xcite . although many argue that outlier dimensions "disrupt" model representations, making them less interpretable and hindering model performance, ablating outlier dimensions has been shown to cause downstream performance to decrease dramatically @xcite . 1 code: https://github.com/wrudman/outlier_dimensions there currently is little understanding of how fine-tuning impacts outlier dimensions and why ablating outlier dimensions is harmful to downstream performance. we address this gap in the literature by investigating 1) how fine-tuning changes the structure of outlier dimensions and 2) testing the hypothesis that outlier dimensions contain taskspecific knowledge. this study makes the following novel contributions: ## related works two seminal works discovered the presence of "outlier" @xcite or "rogue" @xcite dimensions in pre-trained llms. following @xcite and @xcite , we define outlier dimensions as dimensions in llm representations whose variance is at least 5x larger than the average variance in the global vector space. the formation of outlier dimensions is caused by a token imbalance in the pre-training data with more common tokens having much higher norms in the outlier dimensions compared to rare tokens @xcite . although the community agrees on the origin of outlier dimensions, their impact on the representational quality of pre-trained llms has been widely contested. the concept of isotropy (i.e., the uniformity of variance in a distribution) is closely related to out-figure 1 : average activation diagrams of sentence embeddings on the sst-2 validation dataset. the x-axis represents the index of the dimension, and the y-axis is the corresponding magnitude in that given dimension. top: pre-trained models where no fine-tuning occurs. bottom: models fine-tuned to complete sst-2. lier dimensions. namely, the presence of outlier dimensions causes model representations to be highly anisotropic @xcite . many previous works have argued that mitigating the impact of outlier dimensions by forcing llm representations to be isotropic improves model interpretability and performance @xcite @xcite @xcite . further, timkey and van schijndel (2021) claim that outlier dimensions do not meaningfully contribute to the model decision-making process and that removing outlier dimensions aligns llm embeddings more closely to human similarity judgments. although the notion that isotropy is beneficial to model representations has been widely adopted in the literature, recent studies have shown that many tools used to measure isotropy and the impact of outlier dimensions in nlp are fundamentally flawed @xcite . there is a growing body of literature arguing that anisotropy is a natural consequence of stochastic gradient descent and that compressing representations into low-dimensional manifold correlates with improved downstream performance @xcite @xcite . recent works in nlp suggest that llms store linguistic and task-specific information in a low-dimensional subspace @xcite @xcite . further, rudman and eickhoff (2023) argue that encouraging the formation of outlier dimensions in llm representations improves model performance on down-stream tasks. in this study, we demonstrate that certain llms store task-specific knowledge in a 1-dimensional subspace and provide evidence supporting claims that outlier dimensions are beneficial to model performance. ## experiments training details we fine-tune 4 transformer encoder llms: bert @xcite , albert @xcite , distilbert @xcite , roberta @xcite and 4 transformer decoder llms: gpt-2 @xcite , pythia-70m, pythia-160m, pythia-410m @xcite . we fine-tune our models on 5 binary classification tasks contained in the glue benchmark @xcite : sst-2 @xcite , qnli @xcite , rte @xcite , mrpc @xcite , qqp. a detailed description of each task is available in section a of the appendix. a detailed description of our hyperparameter search, exact hyperparameters, and random seeds is given in section b. we follow the common practice of reporting results on glue tasks using the hidden validation dataset. ## discussion previous studies investigating the role of outlier dimensions tend to only focus on bert or roberta, yet make broad claims about the role of outlier dimensions for llms as a whole @xcite . our results demonstrate that outlier dimensions have different functions for different models and tasks. in particular, outlier dimensions contain task-specific knowledge that can complete downstream fine-tuning tasks for some models (gpt-2, albert, pythia-160m, and pythia-410m) but not for others (bert, distil-bert, roberta, and pythia-70m). future work should avoid generalizing results from one observation to all other models. although numerous studies have argued that outlier dimensions are harmful to model representations @xcite @xcite , we quantitatively show that the single principal outlier dimension can store enough task-specific knowledge in gpt-2, al-bert, pythia-160m, and pythia-410m to complete downstream tasks. in cases where the principal outlier dimension does not contain sufficient taskspecific knowledge to complete downstream tasks, we find that there are often non-principal outlier dimensions that retain high 1-d performance. in particular, figure 3 shows that there are non-principal outlier dimensions in bert and roberta that can complete qnli using equation 1 with only a 0.56% and 1.01% performance decrease compared to using full model representations and a classification head. these findings help explain recent results showing that encouraging even larger variance in outlier dimensions is beneficial to llm fine-tuning performance @xcite . additionally, our finding that 1-d subspaces contain task-specific knowledge strengthens the arguments that llms store linguistic knowledge in a low-dimensional subspace @xcite @xcite . the persistence of the same small set of outlier dimensions in pre-training and finetuning across various classification tasks and random seeds provides strong evidence that the lowdimensional subspaces learned for different tasks are highly similar. namely, when fine-tuning, certain llms adapt the same small set of outlier dimensions to store task-specific knowledge. ## conclusions & future works this study challenges the dominant belief in the literature that outlier dimensions are detrimental to model performance by demonstrating that 1) the exact outlier dimensions that emerge in pretraining persist when fine-tuning models regardless of the classification task or random seed and 2) in some llms, outlier dimensions contain enough task-specific knowledge to linearly separate points by class label. however, it is still unclear why the principal outlier dimension contains task-specific knowledge in some models and not others. future work should investigate the specifics of these occurrences and how this finding is affected by model scale, architectural choices, and training objectives. ultimately, understanding the mechanisms and implications of outlier dimensions in llms can contribute to advancements in transfer learning, model interpretability, and optimizing performance in several nlp tasks.