Encourage or Inhibit Monosemanticity? Revisit Monosemanticity from a Feature Decorrelation Perspective (2024)

Abstract: To better interpret the intrinsic mechanism of large language models (LLMs), recent studies focus on monosemanticity on its basic units. A monosemantic neuron is dedicated to a single and specific concept, which forms a one-to-one correlation between neurons and concepts. Despite extensive research in monosemanticity probing, it remains unclear whether monosemanticity is beneficial or harmful to model capacity. To explore this question, we revisit monosemanticity from the feature decorrelation perspective and advocate for its encouragement. We experimentally observe that the current conclusion by (CITATION), which suggests that decreasing monosemanticity enhances model performance, does not hold when the model changes. Instead, we demonstrate that monosemanticity consistently exhibits a positive correlation with model capacity, in the preference alignment process. Consequently, we apply feature correlation as a proxy for monosemanticity and incorporate a feature decorrelation regularizer into the dynamic preference optimization process. The experiments show that our method not only enhances representation diversity and activation sparsity but also improves preference alignment performance.

## introduction recent years have witnessed significant breakthroughs made by large language models (llms), which demonstrate impressive performance across a wide range of nlp tasks @xcite @xcite . meanwhile, understanding how they iteratively develop and refine suitable representations from inputs remains opaque @xcite @xcite . mechanistic interpretability is to understand neural networks by breaking them into components that are more easily understood than the entire network @xcite @xcite . however, the neuron, the most basic computational unit of the neural network, is not a natural unit for human understanding. this is because many neurons are polysemantic, responding to mixtures of seemingly unrelated inputs @xcite @xcite . towards fundamental interpretability, very recent works study the monosemantic neurons: those form a one-to-one correlation with their related input features @xcite @xcite . researchers in openai have applied the sparse autoencoder @xcite with dictionary learning to identify the monosemanticity at a large scale. given the computational cost in training sparse autoencoder and the human labor required for generating interpretations, their detailed interpretability is specifically focused on 4,096 features @xcite . furthermore, the studies by @xcite and @xcite proposed efficient monosemanticity proxies, offering a pathway for the exploration of this model property. despite success, the relationship between monosemanticity and llm's capacity (such as robustness and alignment), remains a subject of ongoing debate. it raises an open question: should monosemanticity be encouraged or inhibited for llm's alignment? to tackle the aforementioned challenges, in this paper, we revisit monosemanticity from the perspective of feature decorrelation and show a positive correlation between monosemanticity and within-model capacity. consequently, we demonstrate this experimentally and propose a decorrelation regularization approach to enhance monosemanticity. specifically, the main contributions of this paper are summarized as follows: ## monosemanticity definition to avoid confusion caused by terminology, we first clarify the definitions of the terms concept, feature, and neuron in this context. ## monosemanticity proxy due to the challenges of identifying monosemanticity on a large scale, researchers have proposed approximate methods to estimate monosemanticity @xcite . following common practices in transformer interpretability, these studies focus on the activations from multi-layer perceptrons (mlps) because of their crucial role in preserving concept-level knowledge @xcite . ## decorrelation regularizer enhances monosemanticity the positive correlation between monosemanticity and model alignment performance motivates us to enhance monosemanticity. given that feature decorrelation is a proxy for monosemanticity and tractable, we propose to apply the l @xmath0. it penalizes the 0.0 0.2 0.4 0.6 0.8 1.0 relative layer depth 2 1 0 1 relative improvement (%) model gpt2-medium(355m) gpt2-large(774m) gpt2-xl(1.5b) gpt-neo(2.7b) figure 3: relative changes of normalized median (||w in || 2 b in ), a proxy for monosemanticity, across different gpt2 models after dpo training. 0.0 0.2 0.4 0.6 0.8 1.0 relative layer depth 0.0 0.2 0.4 0.6 0.8 1.0 feature decorrelation base_wiki dpo_wiki base_cog dpo_cog dpo_syco base_syco a well-trained dpo significantly increases feature decorrelation, i.e., the proxy for monosemanticity. the drop in later layers has also been observed in @xcite , attributed to their proximity to the supervision signal. frobenious distance between the feature correlation matrix zz t and the identity matrix i (fully decorrelated). this regularizer is widely adopted in self-supervised learning to encourage feature diversity and prevent dimensional feature collapse @xcite @xcite . we incorporate this regularizer to the original dpo training objective and set the weight for this term as 0.0001. we name this method as decorrelated policy optimization (decpo). ## monosemanticity contributes to preference optimization the previous section has provided evidence that a decorrelation regularizer can enhance monosemanticity. now, we continue to validate our hypothesis, monosemanticity should be encouraged, by evaluating whether decpo will boost alignment performance. although decorrelated representations have been widely discussed in both computation vision and language processing @xcite , limited research has examined this issue within existing preference optimization algorithms, such as dpo @xcite and proximal policy optimization (ppo) @xcite . ## conclusion in this paper, we have revisited recent studies in monosemanticity probing and proposed a monosemanticity proxy via feature decorrelation perspective. to study the research question should monosemanticity be encouraged or inhabited in a model level for alignment training? we experimentally provide the empirical evidence that the alignment, such as dpo, can improve monosemanticity. we have also clarified that there is no clear relation between the monosemanticity degree and model size. then, we have studied the effects of enhanced monosemanticity via applying a decorrelation reg-ularizer in dpo training. the evidence from the better alignment experiment further verifies our hypothesis that monosemanticity should be encouraged for better model capacity.