Packing Analysis: Packing Is More Appropriate for Large Models or Datasets in Supervised Fine-tuning (2025)

Abstract: Packing, initially utilized in the pre-training phase, is an optimization technique designed to maximize hardware resource efficiency by combining different training sequences to fit the model’s maximum input length. Although it has demonstrated effectiveness during pre-training, there remains a lack of comprehensive analysis for the supervised fine-tuning (SFT) stage on the following points: (1) whether packing can effectively enhance training efficiency while maintaining performance, (2) the suitable size of the model and dataset for fine-tuning with the packing method, and (3) whether packing unrelated or related training samples might cause the model to either excessively disregard or over-rely on the context.In this paper, we perform extensive comparisons between SFT methods using padding and packing, covering SFT datasets ranging from 69K to 1.2M and models from 8B to 70B. This provides the first comprehensive analysis of the advantages and limitations of packing versus padding, as well as practical considerations for implementing packing in various training scenarios. Our analysis covers various benchmarks, including knowledge, reasoning, and coding, as well as GPT-based evaluations, time efficiency, and other fine-tuning parameters. We also open-source our code for fine-tuning and evaluation and provide checkpoints fine-tuned on datasets of different sizes, aiming to advance future research on packing methods.

## introduction supervised fine-tuning (sft) refers to adapting a pre-trained model to perform specific tasks by train-ing it on a labeled conversation dataset consisting of (instruction, answer) pairs @xcite @xcite @xcite @xcite @xcite . as models and datasets grow, the cost of fine-tuning rises. identifying cost-effective methods, optimizing resource utilization, and alleviating the financial burden of large-scale training present new challenges in the sft process @xcite @xcite . packing addresses these challenges by combining multiple training samples into a single sample. originally used during the pre-training phase, packing extends each training sequence to the model's maximum input length, optimizing hardware resource usage, such as gpus, and improving training efficiency @xcite @xcite @xcite . despite its proven effectiveness during the pre-training phase, for sft, a thorough analysis is still lacking on: (1) whether packing can effectively enhance training efficiency while maintaining performance, (2) the suitable size of the model and dataset for fine-tuning with packing, and (3) whether packing unrelated or related training samples might cause model to either excessively disregard or over-rely on the context. to address these concerns, this paper provides a thorough analysis of packing during the supervised fine-tuning (sft) stage. specifically, we perform extensive comparisons between sft methods using padding and packing, covering sft datasets ranging from 69k to 1.2m and models from 8b to 70b. our comparisons include various benchmarks, such as knowledge, reasoning, and coding, gpt-based evaluations, time efficiency, and other fine-tuning parameters, concluding that: ## related work supervised fine-tuning (sft) in large language models (llms) involves additional training on a dataset of (instruction, answer) pairs. this approach helps align the llms' training goal of predicting the next word with users' expectations for the models to follow human instructions more accurately @xcite @xcite @xcite @xcite @xcite @xcite . the initial step in sft is to create annotated data, but current sft datasets are often constrained by their limited quantity, diversity, and creativity @xcite @xcite @xcite . to address this issue and rich resources for research, in one line, researchers distilled data from powerful large models (e.g., gpt-4 (openai, 2023)) @xcite @xcite . on the other line, some researchers are working on methods to enable pretrained models to self-generate useful sft data @xcite . once highquality sft datasets are created, the next step is to use them for fine-tuning pre-trained models. many studies are dedicated to minimizing the costs of fine-tuning, including gpu usage and time, while maintaining performance, such as light-weight finetuning @xcite , speeding up attention algorithm for transformerbased llms @xcite , and efficient distributed fine-tuning @xcite 2 . in this paper, we are analyzing one of the efficient fine-tuning techniques: packing, which packs multiple training samples into a single sample to maximize the utilization of hardware resources and enhance fine-tuning efficiency, providing a comprehensive understanding of its effectiveness and potential risks. below, we will thoroughly analyze its effectiveness and potential risks. ## methods: padding and packing padding and packing are two distinct methods to organize training samples, shown in figure 1 . in this section, we first define a set of mathematical symbols, followed by a detailed explanation of one padding method and two packing methods: random packing and greedy packing. we assume that {c} = {c 1 , . . . , c n } denotes the training conversations, where n is the size of the training conversation, and c @xmath0, y i 1 ), . . . , (x i m , y i m )}, @xmath1. in this paper, we also use the terms "single-turn conversation" and "multi-turn conversation" to distinguish between training conversations that consist of a single instruction and answer pair (@xmath2. ## experimental setups in this section, we sequentially describe our "training setups" in 4.1 and "evaluation setups" in 4.2. ## results and analysis in this section, we provide our experimental results as well as analysis based on them. ## conclusion in this paper, we conduct a thorough comparison of sft methods using padding and packing, analyzing datasets from 69k to 1.2m and models ranging from 8b to 70b. this provides the first detailed examination of the advantages and limitations of packing versus padding, as well as practical considerations for implementing packing in various training scenarios. our evaluation spans a range of benchmarks, including knowledge, reasoning, and coding, and includes gpt-based assessments, time efficiency, and other fine-tuning factors. we also open-source our code, along with checkpoints fine-tuned on datasets of varying sizes, to support future research into packing techniques.