Constructing Your Model’s Value Distinction: Towards LLM Alignment with Anchor Words Tuning (2025)

Abstract: With the widespread applications of large language models (LLMs), aligning LLMs with human values has emerged as a critical challenge. For alignment, we always expect LLMs to be honest, positive, harmless, etc. And LLMs appear to be capable of generating the desired outputs after the alignment tuning process, such as the preference tuning via reinforcement learning from human feedback (RLHF). However, it also raises a question about **after alignment, do LLMs genuinely obtain a value distinction between positives and negatives, beyond the generation of positive outputs?** In this work, we start by investigating this question from the token distribution perspective. Our findings reveal that compared to the unaligned versions, LLMs after alignment exhibit a larger logits gap between positive and negative tokens at each generation step, which suggests that LLMs do obtain a value distinction of positives and negatives after alignment. Meanwhile, it also motivates us to achieve alignment by directly constructing such value distinction, thus alleviating the excessive reliance on computational resources required by training-time alignment. Specifically, we propose a representation editing method that intervenes the last hidden representation by amplifying the logits difference between positive and negative tokens (defined as anchor words). Experimental results demonstrate that the proposed method not only achieves effective alignment, but also requires fewer computational resources compared to training-time alignment methods

## introduction large language models (llms) such as chatgpt @xcite and @xcite , which are pretrained on extensive datasets, have demonstrated impressive abilities across numerous tasks. however, when applying these aligned llms understanding value distinction achieve alignment … … generation positive ≫ negative positive ≠ negative vanilla llms to real world scenarios, they may accidentally output toxic and harmful responses due to the complex nature of the training data. with the advancement of llm researches and applications, it is foreseeable that aligning llms with human values will become increasingly crucial and challenging @xcite . fortunately, relevant research efforts, such as rlhf @xcite and dpo @xcite , have achieved remarkable improvements on llm alignment by fine-tuning models using human preference datasets. through preference learning, llms after alignment demonstrate the capability to produce text that exhibits enhanced alignment with human preferences across both affective and contextual dimensions. however, just like how children can exhibit certain abilities they do not inherently possess by imitating adult speech patterns, llms may also simply replicate specific generation patterns from preference datasets to fulfill the alignment expectations of trainers @xcite . intrigued by this interesting and critical question, our work starts by investigating whether an aligned llm, beyond generating positive responses, has genuinely developed a form of value judgment (e.g., a value distinction between positives and negatives) 2 . specifically, we delve into aforementioned problem from two perspectives: llm understanding and generation. for understanding, we leverage a question set which consists of 100 harmful and 100 harmless queries for llms @xcite and employ principal component analysis (pca) to visualize the hidden representations of the last input tokens output by the top model layers. notably, previous studies @xcite have explored the differences in hidden states between aligned llms with and without safety prompts, but here, our focus is primarily on the differences between unaligned and aligned llms. the visualization results indicate that in models' representation space, harmful and harmless questions can be easily distinguished in aligned llms, whereas in unaligned llms, these two types of queries remain intermixed, which suggests that, from the perspective of llm understanding, alignment allows llms to more clearly distinguish between harmful and harmless queries. naturally, with a clear understanding of what is positive and negative, llms are expected to hold such value judgment to produce positive outputs in subsequent generations. however, is this truly the case? to further figure out the answer, we then turn our attention to the generation process. for generation, we manually select 7 positive and 7 negative words and compute the average difference in logits between the two sets at each generation step. experimental results show that in aligned llms, the difference between positive and negative words at each generation step is about 50 times greater than that in unaligned versions where the difference in unaligned models is approximately 0, suggesting that although aligned llms do not explicitly "speak" these positive words at every step, they "know" that positive content is more appropriate to be generated during the whole generation process. in other words, llms after alignment appear to obtain a certain value distinction between positives and negatives. inspired by these findings, we propose a testtime alignment method, named awot (anchor words tuning), which constructs aforementioned value distinction by amplifying the logits difference between positive and negative tokens (i.e., the anchor words). during test time, awot tunes the representations of the last model layer to maximize the logits difference through gradient-based optimization, which means that unaligned llms are intrinsically intervened rather than just at the prompt format level, such as prompt engineering, to construct such value distinction. on the other hand, one of the main demerits of training-time alignment method is the high computational demands and time cost. in contrast, since awot operates only on the final layer's representation at test time, it is more lightweight and computationally efficient. experimental results demonstrate both the effectiveness and efficiency of the proposed method. the main contributions of our work can be summarized as follows: ## exploring the value distinction in llms in this section, we explore whether llms obtain a distinction between positive and negative values after alignment, beyond the generation of text that seemingly meets the requirements. firstly, we delve into the models' understanding of harmful and harmless questions in the representation space. previous studies @xcite , for each model, we plot two groups of points: with harmful and harmless questions. it can be observed that harmful and harmless questions can be clearly distinguished by aligned llms, whereas vanilla llms exhibit significant entanglement, struggling to differentiate the two opposing queries. 2024c; @xcite have clarified the model mechanisms by probing the inner representations of llms, motivating us to closely examine the representations of positive and negative queries output by llms to investigate their ability in distinguishing various values. ## constructing the model's value distinction motivated by the findings in preliminary experiments, in this section, we seek to achieve alignment by constructing the model's value distinction. the method consists of i) the anchor words selection module that obtains positive and negative words through two kinds of operations: manually and automatically, ii) and the representation tuning module that optimizes the hidden states to maximize the value distinction. ## conclusion in this work, we first study the behavioral differences of llms before and after alignment. from the perspectives of understanding and generation, we reveal that llms appear to obtain a distinction between positive and negative values through alignment. and they still hold such value distinction during the generation process, although experiments show that they tend to gradually forget the value distinction as the generation step increases. inspired by these findings, we propose to achieve alignment by constructing such value distinction. the proposed method awot adjusts the model representation to maximize the value distinction during inference. as a test-time alignment method, awot requires less computational resources and time overhead. extensive experiments demonstrate the effectiveness and efficiency of proposed method.