How Abilities in Large Language Models are Affected by Supervised Fine-tuning Data Composition (2024)

Abstract: Large language models (LLMs) with enormous pre-training tokens and parameters emerge diverse abilities, including math reasoning, codegeneration, and instruction following. These abilities are further enhanced by supervised fine-tuning (SFT). While the open-source community has explored ad-hoc SFT for enhancing individual capabilities, proprietary LLMs exhibit versatility across various skills. Therefore, understanding the facilitation of multiple abilities via SFT is paramount. In this study, we specificially focuses on the interplay of data composition between mathematical reasoning, code generation, and general human-aligning abilities during SFT. We propose four intriguing research questions to explore the association between model performance and various factors including data amount, composition ratio, model size and SFT strategies. Our experiments reveal that distinct capabilities scale differently and larger models generally show superior performance with same amount of data. Mathematical reasoning and code generation consistently improve with increasing data amount, whereas general abilities plateau after roughly a thousand samples. Moreover, we observe data composition appears to enhance various abilities under limited data conditions, yet can lead to performance conflicts when data is plentiful. Our findings also suggest the amount of composition data influences performance more than the composition ratio. In analysis of SFT strategies, we find that sequentially learning multiple skills risks catastrophic forgetting. Our proposed Dual-stage Mixed Fine-tuning (DMT) strategy offers a promising solution to learn multiple abilities with different scaling patterns.

## introduction recent research has demonstrated the remarkable and versatile proficiency of large language models (llms) in dealing with a variety of real-world * work done during internships at alibaba group. however, existing research has mostly conducted separate sft investigations on each of the three tasks, where reasoning and coding abilities require sft on in-domain human-annotated or augmented data @xcite @xcite while diverse and complex human instructions are applauded for aligning human intentions @xcite @xcite @xcite @xcite . as shown by the strong performance of proprietary llms such as @xcite and claude, llms have the potential to master all the tasks in one model. therefore, it is of paramount importance to investigate the versatile performance of sft with composite task data, and understanding and addressing the challenges posed by the data composition problem in the sft stage is crucial for further enhancing the capabilities of llms in a comprehensive manner. in essence, the tasks of reasoning, coding, and aligning human intentions are of different characteristics. reasoning and coding tasks require ad-hoc abilities of complex and detailed logic in decomposing task instructions and dealing with nonlinguistic and symbolic features @xcite , whereas aligning human intentions requires versatility and understanding obscure intentions expressed in human instructions @xcite . given the fundamental difference among the tasks, multi-task learning with composite data fine-tuning for small-scaled pre-trained language models is prone to catastrophic forgetting @xcite , hindering the fine-tuned performance of one model on separate tasks. many efforts have been made to compensate for the phenomenon @xcite @xcite . there has also been research discovering that scaling up the pre-trained language model scale and the fine-tuning data scale are beneficial for zero-shot out-of-domain generalization on various linguistic tasks while leaving out the assessment of in-domain performance @xcite @xcite . given the increased capacity of llms, the multi-task performance by sft on composite data of essentially different downstream tasks is less studied. understanding the sft performance with composite data and corresponding scaling patterns is of great utility in practice. in this study, we focus on the data composition problem among mathematical reasoning, code generation, and general human-aligning abilities in sft. we aim to comprehensively investigate the relationship between model performance and different factors including data amount, data composition ratio, model scales, and sft training strategies. we also investigate how the relationship varies under different scales. specifically, we focus on the following four research questions: to answer these questions, we conduct experiments on three benchmarks, which are gsm8k @xcite for mathematical reasoning, humaneval @xcite for coding, and mt-bench @xcite for general human alignment. we fine-tune llms on the related training data to activate these abilities. furthermore, we conduct extensive analysis regarding model parameter scales ranging from llama 7b to 33b @xcite and explore four different sft strategies shown in figure 1 : multi-task learning, sequential training, mixed sequential training, and dual-stage mixing fine-tuning (dmt), providing empirical guidance for learning a versatile llm with composite sft. the key findings of this paper can be summarized as follows: while research has probed into the generalization capabilities of llms within out-of-distribution domains @xcite @xcite , the effects of multi-task training on in-domain performance remain under-explored. with the ascent of proprietary models like chat-gpt, the focus on sft for aligning llms with human intent has intensified @xcite . moving away from crowd-sourced sft data, recent initiatives have generated sft datasets from user logs within proprietary llm platforms @xcite , employing the models themselves to assist in the data generation process @xcite @xcite @xcite . additionally, methods to improve sft data quality have been proposed, targeting more accurate alignment with human interactions @xcite @xcite . furthermore, sft has proven beneficial for @xcite . this paper examines the sft performance using composite datasets, considering different model sizes and data amounts. ## experiments we have sft datasets {d 1 , d 2 , ..., d k } where each d @xmath0,j , r i,j } j contains queries and responses from one source. we consider each sft dataset to correspond to one ability and we also have k in-domain metrics to measure them. we investigate the performances of in-domain metrics with different dataset compositions (d ⊂ ∪ 1≤@xmath1. ## conclusion we explore the data composition in the sft phase, focusing on mathematical reasoning, code generation, and general human-aligning abilities. we formulate four research questions to guide our investigation and analyze the scaling trends between different abilities and factors (e.g. data amount, data ratio, model parameters, and training strategies). our findings reveal distinct scaling patterns among different abilities, with larger models demonstrating superior performance when trained with the same amount of data. moreover, mixing data sources in the sft phase improves performance in low-resource scenarios but diminishes in highresource scenarios. interestingly, the phenomenon of low-resource gain becomes more prominent as the model parameter size increases. furthermore, our observations indicate that data amount directly influences performance conflicts, whereas the impact of data ratio is insignificant within our experimental setup. finally, regarding the sft strategies, we demonstrate our proposed dmt strategy effectively alleviates performance conflicts, offering a promising solution to activate multiple abilities.