Extracting and Understanding the Superficial Knowledge in Alignment (2025)

Abstract: Alignment of large language models (LLMs) with human values and preferences, often achieved through fine-tuning based on human feedback, is essential for ensuring safe and responsible AI behaviors. However, the process typically requires substantial data and computation resources. Recent studies have revealed that alignment might be attainable at lower costs through simpler methods, such as in-context learning. This leads to the question: Is alignment predominantly superficial? In this paper, we delve into this question and provide a quantitative analysis. We formalize the concept of superficial knowledge, defining it as knowledge that can be acquired through easily token restyling, without affecting the model’s ability to capture underlying causal relationships between tokens. We propose a method to extract and isolate those superficial knowledge from aligned models, focusing on the shallow modifications to the final token selection process. By comparing models augmented only with superficial knowledge to fully aligned models, we quantify the superficial portion of alignment. Our findings reveal that while superficial knowledge constitutes a significant portion of alignment, particularly in safety and detoxification tasks, it is not the whole story. Tasks requiring reasoning and contextual understanding still rely on deeper knowledge. Additionally, we demonstrate two practical advantages of isolated superficial knowledge: (1) it can be transferred between models, enabling efficient offsite alignment of larger models using extracted superficial knowledge from smaller models, and (2) it is recoverable, allowing for the restoration of alignment in compromised models without sacrificing performance.

## introduction recent years have witnessed significant advancements of large language models (llms) in various tasks @xcite @xcite . although llms acquire extensive world knowledge, they meanwhile cast serious risks to the society. for example, llms are easily prompted to generate toxic, misleading, or harmful content @xcite @xcite . to ensure that the behaviors of llms adhere to human values and preferences, aligning llms to follow instructions based on human feedback @xcite @xcite @xcite is essential. to obtain satisfactory alignment, the tuning of an llm usually demands a non-trivial amount of data and computation resources. despite the considerable efforts invested in tuning llms @xcite , it has been surprisingly discovered that alignment might be attainable at lower costs or through simpler methods @xcite @xcite . for example, using only a few selected training examples can significantly improve alignment performance, approaching levels achieved through extensive tuning. furthermore, urial @xcite found that alignment often results in "stylistic token shifts," and by employing in-context learning (icl) @xcite with a few restyling examples, alignment can be improved without any further tuning. these findings give rise to the superficial alignment hypothesis @xcite , which suggests that a model may acquire most of its knowledge and abilities during pre-training, while alignment primarily involves superficial adjustments. however, current methods support this hypothesis primarily through informal observations and indirect implications (i.e., because alignment can be achieved through superficial methods, it is hypothesized to be superficial). there remains a lack of rigorous, deep analysis regarding the extent to which alignment relies on superficial knowledge and whether alignment is purely superficial. to fill this gap, we first formalize the previously vague concept of superficial knowledge. we define superficial knowledge as the type of knowledge that can be easily acquired through simple token restyling, without requiring modifications to the model's understanding of the underlying causal relationships between tokens and the process of knowledge extraction and compression. in contrast, deep knowledge pertains to the model's ability to capture token relationships and extract meaningful insights from the data. we propose a method to extract and isolate superficial knowledge from the alignment process. to ensure the extracted knowledge remains superficial, we restrict our modifications to shallow, simple structures -specifically, the linear projection head of the llm. this affects only the final token selection process, without altering the intermediate token merging or self-attention mechanisms. by doing so, we avoid disrupting the deep knowledge associated with internal token interactions. furthermore, to ensure that no new knowledge is introduced into the model and to focus exclusively on analyzing the knowledge derived from alignment, we employ distillation to finalize the extraction process. with the extracted and separated superficial knowledge, we can quantify the superficial portion of alignment by comparing the aligned model with a base model augmented only with superficial knowledge across benchmarks in math, safety, toxicity, and truthfulness. our key findings are twofold: in addition, since our extracted superficial knowledge is stored in a simple and modular structure, we have also discovered several useful properties of superficial knowledge. we further demonstrate the superficial advantage (sa)-the benefits of isolating superficial knowledge alone. sa1: weak-to-strong superficial alignment. our experiments reveal that the extracted superficial knowledge is transferable across models. this transferability can be leveraged for offsite alignment of larger models-superficial knowledge extracted from a smaller, weaker model can be applied to a larger, stronger model. this allows for plug-and-play alignment of the larger model without requiring extensive tuning. sa2: recoverable superficial safety. previous work @xcite has shown that safety mechanisms can be easily compromised, such as through slight fine-tuning on as few as 10 samples. however, with our extracted superficial knowledge, we can re-attach the lightweight structure encapsulating this knowledge to a de-aligned llm and successfully recover 88% of the alignment effects without compromising mmlu accuracy. ## conclusion in this paper, we propose a method to separate superficial knowledge from deep knowledge within alignment, enabling us to quantify the the superficial portion of alignment. our analysis finds that superficial knowledge indeed constitutes a large proportion of alignment, though not entirely. knowledge beyond the superficial, related to reasoning abilities and contextual integration, is also crucial to alignment. additionally, our extracted superficial knowledge extends beyond mere analytical use, offering practical applications such as weak-tostrong superficial alignment and recovering compromised safety.