Mitigating Training Imbalance in LLM Fine-Tuning via Selective Parameter Merging (2024)

Abstract: Supervised fine-tuning (SFT) is crucial for adapting Large Language Models (LLMs) to specific tasks. In this work, we demonstrate that the order of training data can lead to significant training imbalances, potentially resulting in performance degradation. Consequently, we propose to mitigate this imbalance by merging SFT models fine-tuned with different data orders, thereby enhancing the overall effectiveness of SFT. Additionally, we introduce a novel technique, “parameter-selection merging,” which outperforms traditional weighted-average methods on five datasets. Further, through analysis and ablation studies, we validate the effectiveness of our method and identify the sources of performance improvements.

## introduction thanks to the substantial expansion of training scale and model size, large language models (llms) have achieved significant breakthroughs across a broad spectrum of nlp tasks @xcite . for downstream tasks, supervised fine-tuning (sft) is a crucial technique for llms, enabling the customization of pretrained models for specialized tasks and domains @xcite . the sft process typically involves a few iterations of training on task-specific data. while existing research generally assumes that the order of training samples has a negligible impact on final model performance, or that sufficient iterations can mitigate any potential effects, our preliminary investigations suggest otherwise. we found that the position of sft training samples significantly affects their final training outcomes. for instance, figure 1 ## experiments this section presents the experimental results. detailed descriptions of the datasets and evaluation metrics employed are provided in the appendix, under section b. ## conclusion this study reveals the overlooked negative impact of training data order on llm fine-tuning, which can result in significant training imbalances, and proposes mitigating these imbalances by merging models fine-tuned with diverse data orders. furthermore, it introduces a novel and effective parameter merging technique, "parameter-selection merging." the efficacy of parameter-selection method suggests that it is not necessary to incorporate information from all sub-models at each parameter dimension in parameter merging. this discovery broadens the research landscape for parameter merging, opening up new avenues for future investigations.