From Language Modeling to Instruction Following: Understanding the Behavior Shift in LLM s after Instruction Tuning (2024)

Abstract: Large Language Models (LLMs) have achieved remarkable success, where instruction tuning is the critical step in aligning LLMs with user intentions. In this work, we investigate how the instruction tuning adjusts pre-trained models with a focus on intrinsic changes. Specifically, we first develop several local and global explanation methods, including a gradient-based method for input-output attribution, and techniques for interpreting patterns and concepts in self-attention and feed-forward layers. The impact of instruction tuning is then studied by comparing the explanations derived from the pre-trained and instruction-tuned models. This approach provides an internal perspective of the model shifts on a human-comprehensible level. Our findings reveal three significant impacts of instruction tuning: 1) It empowers LLMs to recognize the instruction parts of user prompts, and promotes the response generation constantly conditioned on the instructions. 2) It encourages the self-attention heads to capture more word-word relationships about instruction verbs. 3) It encourages the feed-forward networks to rotate their pre-trained knowledge toward user-oriented tasks. These insights contribute to a more comprehensive understanding of instruction tuning and lay the groundwork for future work that aims at explaining and optimizing LLMs for various applications. Our code and data are publicly available at https://github.com/JacksonWuxs/Interpret_Instruction_Tuning_LLMs.

## introduction the strong capability of large language models (llms) to align with user intentions is wellrecognized across various real-world applications, where they are expected to be helpful, honest, and harmless ai assistants @xcite . central to these roles, being "helpful" is the most fundamental requisite, emphasizing that llms should help users to complete various tasks, known as the "instruction following" capability. many studies @xcite show that instruction tuning, also called supervised fine-tuning, is critical to acquire such capability, by fine-tuning pre-trained models on high-quality prompt-response pairs. however, the impact of instruction tuning on the helpfulness of language models remains largely unexplored, limiting the development of better ai assistants. in this work, we focus on exploring how instruction tuning changes pre-trained models. specifically, how do instruction-tuned models utilize the instruction words to guide their generation in a way that differs from pre-trained models? step further, how do self-attention heads and feed-forward networks contribute to this difference by adapting their pre-trained knowledge, respectively? however, technically answering these questions by interpreting llms is non-trivial. for the first question, we aim to quantify the importance of prompt words to response words, known as attribution explanations. existing work @xcite @xcite is proposed for the classification problems, which is not suitable for auto-regressive llms. for the second question, we seek to interpret both selfattention and feed-forward layers within llms. a straightforward method @xcite , which projects weight vectors into the word embedding space and then selecting the most activated words as explanations, is compromised by the polysemic nature of model weights @xcite , leading to unclear and not concise explanations. other researchers have studied internal activations of the models, such as heatmap visualization @xcite , sparse autoencoder decomposition @xcite , and knowledge probing @xcite , while they may yield biased explanations due to the potential bias in the chosen samples for collecting activations. overall, existing explanation methods cannot be directly applied to auto-regressive llms. to fill these gaps, we first develop a series of explanation methods as a toolbox to study llms, including a gradient-based method for prompt-response attributions, and techniques to interpret the patterns and concepts in self-attention heads and feed-forward networks at a humanunderstandable level. we then investigate the impact of instruction tuning by comparing the explanations coming from the pre-trained and instruction-tuned models. this approach provides an internal perspective of exploring instruction tuning, distinguishing it from existing research that primarily focuses on comparing the performance of the model trained under different settings @xcite @xcite . we obtain three main findings of the impact of instruction tuning as follows: this study reveals that instruction words are crucial to instruction-tuned models because of their consistent impact on the generation process, and further emphasizes the distinctive contributions of self-attention mechanisms and feed-forward networks to this functionality. while our focus is on behavior shifts after instruction tuning, future research might also apply our toolbox to understand llms for various other purposes. ## impact of user prompts for human alignment this section focuses on the differential functionalities of user prompts between the instruction-tuned and pre-trained models. we introduce a gradientbased attribution approach in sec. 4.1 to measure the importance of individual input words on specific output words. in sec. 4.2, we compare word importance densities across various models to study their distinctions in using user prompts. ## shift within instruction-tuned models this section studies the distinctive contributions of components in llms for human alignment. the self-attention heads and feed-forward networks are discussed in sec. 5.1 and 5.2, respectively. ## discussion our findings provide a unique perspective to align with recent studies. 1) the importance of prompt diversity is highlighted by both us and @xcite . since our three findings suggest that instruction tuning links the pre-trained model to user tasks, we could expect a better alignment with human intentions if the model is exposed to broader prompts. ## conclusion this paper presents a comprehensive analysis of instruction tuning for user intention alignment by quantitatively and qualitatively comparing the posthoc explanations between pre-trained and finetuned models. our findings indicate that instruction tuning links the pre-trained model to user intentions, including encoding more instruction words' knowledge within self-attention, and rotating general knowledge from feed-forward networks towards user usage. it is worth mentioning that the interpretability toolbox used in this study can also support future general research on llms.