Learning or Self-aligning? Rethinking Instruction Fine-tuning (2024)

Abstract: Instruction Fine-tuning (IFT) is a crucial phase in building large language models (LLMs). Previous works mainly focus on the IFT’s role in the transfer of behavioral norms and the learning of additional world knowledge. However, the understanding of the underlying mechanisms of IFT remains significantly limited. In this paper, we design a knowledge intervention framework to decouple the potential underlying factors of IFT, thereby enabling individual analysis of different factors. Surprisingly, our experiments reveal that attempting to learn additional world knowledge through IFT often struggles to yield positive impacts and can even lead to markedly negative effects. Further, we discover that maintaining internal knowledge consistency before and after IFT is a critical factor for achieving successful IFT. Our findings reveal the underlying mechanisms of IFT and provide robust support for some very recent and potential future works.

## introduction the advent and evolution of large language models (llms) have marked a significant milestone in natural language processing (nlp) @xcite . as one of the core steps in the construction of llms, instruction fine-tuning (ift) employs supervised instructionresponse pairs to fine-tune llms, thereby facilitating the transformation of llm from a continuous writing model to a question-answering agent @xcite @xcite . despite the crucial role of ift in the construction of llms, there is a significant lack of in-depth research on the mechanisms by which ift operates. in traditional machine learning, supervised learning aims to fit models to specific tasks and data distributions @xcite , whereas the impact of ift on llms is markedly different. as shown in figure 1 , on one hand, one of the most apparent effects of ift is its ability to align the output of llms more closely with the latent behavioral norms contained within the ift data, thereby enabling more effective parameter knowledge expression @xcite . on the other hand, many existing studies @xcite @xcite aim to facilitate domain-specific adaptation of llms through ift, by injecting the world knowledge contained in ift data into llms. unfortunately, both the transfer of behavioral norms and the enhancement of domain knowledge are closely coupled with the corpus applied in ift, rendering the analysis of ift's true effects exceedingly challenging. due to the interconnection of these two effects, it is challenging to discern whether the benefits derived are due to the promotion of more effective expression of parameter knowledge or the injection of additional world knowledge. the coupling between the above two factors, along with a lack of in-depth analysis of the ift's mechanism, hampers our comprehension of the effectiveness of ift. this limitation hinders our ability to develop robust strategies for ift data construction, model training, and model evaluation due to insufficient theoretical support. therefore, a thorough and comprehensive analysis of the underlying core factors that drive the effectiveness of ift is crucial for achieving more effective ift. to this end, this paper designs a knowledge intervention framework for analyzing the underlying mechanisms of ift. the main idea of our framework is to control the consistency between the knowledge in ift data and the existing parameter knowledge of llm, in order to decouple the injection of domain knowledge from the transfer of behavioral norms during ift. this allows for a separate analysis of the roles of these two crucial factors. specifically, we first employ in-context learning (icl) @xcite to probe the internal parameter knowledge of llms. building on this, we intervene in the composition of existing parameter knowledge and the newly introduced world knowledge within ift data and then observe the differences in model behavior after ift using different intervention groups. based on the framework, we conduct an in-depth analysis to answer the following two critical research questions (rq): for rq1, we initially discover that significant discrepancies between the world knowledge contained in ift data and the existing parameter knowledge within llm can substantially undermine the model's abilities. performance derived from a set of ift data that contains incorrect world knowledge but aligns with the model's parameter knowledge is significantly better than that from a set containing correct world knowledge but inconsistent with the model's internal parameter knowledge. to dive into this phenomenon, we explicitly supply llms with the world knowledge necessary for answering the instruction, integrated into the context. this strategy allows the model to focus on transforming the output behavioral norms instead of jointly learning the inconsistent world knowledge. we discover that the detrimental effects caused by the inconsistency between parameter knowledge and world knowledge can be significantly mitigated by explicitly providing such self-contained ift data points. these two findings indicate that attempting to introduce world knowledge through ift that is inconsistent with the model's parameter knowledge can severely undermine the model, which suggests that the injection of world knowledge does not lie at the center of a successful ift. for rq2, we further analyze the model performance under varying degrees of consistency between the knowledge contained in ift data and the parameter knowledge in the original llm. we find that while the consistency between the two has a significant impact on model performance, a higher degree of consistency does not necessarily correlate with better model performance. however, our further research reveals a strong correlation between the model's ultimate performance after ift and the consistency of the model's internal knowledge before and after ift. that is to say, for the model after ift, if its responses are more consistent with the responses produced by the original model from in-context learning probing, then the performance of the model after ift is also better. the validity of the finding is independent of whether the test data belongs to the same domain as the training data, and is also unrelated to the original performance of the base llm. this implies that the phenomenon is solely influenced by the consistency of the knowledge before and after ift. furthermore, we discover that using ift data that is either too consistent or too inconsistent with the original parameter knowledge can lead to a divergence in the model's internal knowledge before and after ift, thereby resulting in a decline in performance. our experiments reveal the fundamental role of ift in the construction of llms. essentially, ift is not a supervised, domain-specific learning process, but a process of self-aligning instruction with existing internal knowledge of llms that can be obtained through few-shot in-context learning probing. our findings not only provide robust theo-retical support for the very recently emerged research on self-alignment @xcite and super-alignment @xcite , etc., but also shed light on the future direction of data construction, model training, and model evaluation for ift. meanwhile, recent efforts have focused on achieving automated alignment, such as selfinstruction-tuning @xcite , self-rewarding @xcite and superalignment @xcite . despite repeated validations of their effectiveness, there remains limited understanding of their success. ## knowledge intervention framework during the process of ift, the potential transfer of behavioral norms and the injection of world knowledge are coupled together. consequently, prior research on ift has struggled to distinguish the relative effects of these two. to further investigate the underlying mechanisms of ift, this paper designs a knowledge intervention framework to decouple these two factors. the main idea behind our framework is to control the association between knowledge in ift data and the existing parameter knowledge in llm, thereby managing the degree of potential world knowledge that would be injected during ift. through this, we can decouple the injection of world knowledge and the transfer of behavioral norms, by observing the effects of ift at varying degrees of world knowledge injection. specifically, we select four multi-choice datasets from different domains. for each question in each dataset, we employ in-context learning to probe the internal parameter knowledge for each base llm. then, we construct multiple instruction datasets by adjusting the consistency of the knowledge within ift data and model parameter knowledge for each base model. finally, we analyze the underlying impact of different degrees of world knowledge injection on ift by fine-tuning llms under different settings and examining their performance on the homogeneous, in-domain, and out-of-domain test sets. in the following, we will introduce our knowledge intervention framework in detail. ## exp-i: does learning domain-specific world knowledge matter for ift? in our first group of experiments, we would like to examine how the additional world knowledge in the ift corpus affects llms. to this end, we conduct experiments under three settings including harmonious, incompatible, and self-aligning. by observing the performance discrepancies among these settings, we analyze the effects of injecting world knowledge into the ift. finding 1. when encompassing correct world knowledge, ift data congruent with model parameter knowledge can lead to superior ift outcomes. to show this, we compare the experimental results under two settings: harmonious and incompatible. note that the datasets of both two settings contain correct world knowledge, meaning each query-response pair aligns with correct knowledge. therefore, the core distinction between these two settings lies in whether the entailed world knowledge is consistent with the parameter knowledge of the llm. specifically, training in the harmonious setting only requires learning behavioral norms without the need for learning any additional knowledge, while training in the incompatible setting requires learning both. finding 2. using ift data that aligns with model parameter knowledge yet is erroneous yields better performance than employing those that are correct but incongruent with model parameter knowledge. to further investigate the impact of learning domain-specific world knowledge on ift, we conduct a more direct comparative experiment between the self-aligning and incompatible settings. for each domain and pre-trained llm, the two settings' datasets use identical queries that the model cannot answer correctly under in-context learning and have different responses: the incompatible dataset's responses are correct, reflecting world knowledge, while the self-aligning dataset's responses represent model parameter knowledge, are incorrect. this finding emphatically indicates that injecting additional domain knowledge through ift also fails to bring effective improvements to llms even in homogeneous and in-domain evaluations. conversely, maintaining consistency with model parameter knowledge, that is, without injecting any additional world knowledge through the self-aligning setting, can yield superior results. moreover, this advantage holds true across all evaluations, as well as different model sizes and architectures. the above results demonstrate a significant decline of performance in models fine-tuned using data that contain correct world knowledge but conflict with model parameter knowledge, compared to using consistent ift data aligned with parameter knowledge. this suggests that introducing additional world knowledge through ift, in cases where it is inconsistent with the parameter knowledge, may not yield the benefits we anticipate. therefore, the core role of ift may lie in facilitating the transfer of behavioral norms, rather than injecting additional domain-specific world knowledge. to further validate this conclusion, in the next section, we design a novel method to decouple the conflict knowledge contained in ift data and present further analysis of this issue. to this end, we start with the data of the incompatible setting to construct a dataset with contextualized knowledge. specifically, given an instruction-answer pair in the incompatible group, we employ gpt-3.5 foot_4 to generate the world knowledge that is required to answer the instruction. the knowledge is then concatenated with the original instruction, as well as the answer, to construct an augmented pair. finally, we use the constructed data to fine-tune llms and compare them with the models fine-tuned with vanilla ift. finding 3. ensuring that the model does not learn world knowledge conflicting with parameter knowledge during ift enhances the effectiveness of ift. the results indicate that we should not force the model to learn additional inconsistent knowledge when the world knowledge in ift data is inconsistent with model parameter knowledge. instead, by decoupling knowledge learning and the transfer of behavioral norms, the problems caused by the above knowledge conflicts can be effectively alleviated. this further verifies our observation in section 4 that additional world knowledge injection may be ineffective or even harmful for ift. during ift, the model should focus on transferring the behavioral norms relying on the existing parameter knowledge and the regularity in the ift data, rather than learning additional world knowledge. therefore, for rq1, we conclude: conclusion 1. for ift, there is little, if not even causing additional damage, benefits from the learning of world knowledge incongruent with parameter knowledge. ## exp-iii: is consistency all you need? the above findings appear to suggest a conclusion: for better transfer of behavioral norms, we should employ ift data that completely aligns with model parameter knowledge without any additional world knowledge. to substantiate this hypothesis, we design a new set of experiments. specifically, by adjusting the proportion of samples derived from the incompatible and the selfaligning group, we aim to adjust the ratios of consistency between the world knowledge in the ift data and model parameter knowledge, thereby observing the variations in ift outcomes under different consistency ratios. finding 4. employing ift data that is fully consistent with model parameter knowledge does not necessarily result in optimal performance. figure 4 displays the results on three base models under different consistency ratios across four domains. from the figure, we can see that: 1) the performance of the incompatible group (i.e., figure 2 : the performance of mistral-7b fine-tuned with instruction datasets of varying consistency ratios. each dataset is composed of a mixture of incompatible and self-aligning data, and the consistency ratio represents the proportion of self-aligning samples. note that a consistency ratio of 0 signifies that all data samples are incompatible, whereas a ratio of 1 indicates exclusively self-aligning data. the results of other base models are presented in the appendix c due to page limitations. @xmath0, which aligns with our previous conclusion; 2) conversely, relying solely on ift data that completely aligns with the model's parameter knowledge (i.e., @xmath1; 3) optimal performance is most frequently achieved through a balanced integration of incompatible and self-aligning data. the ideal proportion of this combination varies across different base models and domains. the observations suggest that while there is a significant impact of the consistency between the knowledge within the ift data and the parameter knowledge of the original model on the performance of the fine-tuned model, this consistency is not the fundamental factor influencing ift performance. therefore, a deeper exploration into the mechanisms underlying knowledge consistency is essential to identify the true determinants of ift's effectiveness. ## exp-iv: rethinking consistency: what really matters for ift? building upon this, we calculate the average pearson correlation coefficient across each test set and subsequently compare it with the performance on the same test set of the fine-tuned model. through this experimental analysis, we aim to observe how the knowledge changes induced by ift affect the ultimate effect of ift. to this end, we employ a partial correlation assessment @xcite to analyze the aforementioned pearson correlation and model performance. specifically, we utilize a total of 96 models, which include 72 models from exp-iii, in addition to models fine-tuned with consistency ratios of 0.05 and 0.1, conducting tests on homogeneous, in-domain, and out-of-domain data. to eliminate the potential influence of the base model's performance on the correlation analysis, ensuring that the results only reflect the differences brought about by varying ift data, we treat the base model's performance as a control variable. by employing partial correlation assessment under the constraint of removing this control variable's influence, we analyze the correlation between the pearson correlation coefficients of model predictions' rankings before and after ift and their ultimate performance. please refer to the appendix d for more details about our analysis. finding 5. the consistency of internal knowledge within a model before and after ift is the key factor affecting the performance of the fine-tuned model. figure 3 presents the results of grouped regression analysis conducted on mistral-7b. to further investigate whether this finding is the underlying mechanism responsible for finding 4, we explore the impact of different consistency data on the correlation of model knowledge before and after ift in exp-iii. we analyze the probability distribution of model outputs under different ift data conditions. consistent with the parameter knowledge evidently leads to a divergence in the internal knowledge of the fine-tuned model from that of the original model, thereby impairing the performance of the fine-tuned model. furthermore, ift using data containing world knowledge completely consistent with the parameter knowledge also may result in a divergence in the knowledge distribution between the original and fine-tuned models. specifically, fine-tuning with only consistent ift data may steer the model towards an sharp knowledge distribution, whereas the original model's parameter knowledge exists as a relatively smooth distribution. conversely, using a middle setting that mixes incompatible and self-aligning data allows the optimization process to maintain model parameter knowledge unchanged while preserving the distribution's smoothness, thereby enabling the fine-tuned model outputs to more closely resemble those of the original model, ultimately yielding better performance. the experimental results mentioned above demonstrate that the key to the performance of fine-tuned models lies in the consistency of model parameter knowledge before and after ift. in fact, the core mechanism underlying the superior performance observed in both finding 1 and finding 2 for the ift data is attributed to the maintenance of consistency in model knowledge before and after ift. therefore, for rq2 raised above, we conclude: conclusion 2. the essence of an effective ift lies in maintaining the consistency of model parameter knowledge before and after ift. ## conclusion and discussion our experiments and conclusions indicate that the core function of ift is not to learn domain-specific world knowledge. instead, learning world knowledge that is inconsistent with model parameter knowledge actually undermines the performance of the model in all evaluations from homogeneous, in-domain to out-of-domain. furthermore, we discover that the consistency of model parameter knowledge before and after ift (i.e., the knowledge probed through in-context learning before ift and the knowledge exhibited under zero-shot setting after ift) plays a crucial role in determining the ultimate performance of the fine-tuned model. these two findings unveil a fundamental mechanism of ift, that is, ift is not a supervised, domainspecific knowledge learning process, but a process of self-aligning instruction with the already existing parameter knowledge of llms. therefore, the ultimate determinant of ift effect is not the extent of domain knowledge injection, but rather whether the ift process can facilitate more effective selfaligning, thereby enhancing the expression of the model's parameter knowledge under the zero-shot question-answering paradigm after ift. our discovery not only provides guidance for future ift data construction, model training, and model evaluation but also provide robust support for some very recent studies. for instance, superalignment @xcite aims to use a weak model to guide a strong model's alignment. our conclusion proves that it is entirely possible to use a weak model with less knowledge to guide a strong, more knowledgeable model for ift. our conclusions elucidate the viable underlying factors for self-instruction-tuning @xcite , self-rewarding @xcite and consistent alignment @xcite and provide a solid foundation for the future development of these studies.