Threshold Filtering Packing for Supervised Fine-Tuning: Training Related Samples within Packs (2025)

Abstract: Packing for Supervised Fine-Tuning (SFT) in autoregressive models involves concatenating data points of varying lengths until reaching the designed maximum length to facilitate GPU processing. However, randomly concatenating data points can lead to cross-contamination of sequences due to the significant difference in their subject matter. The mainstream approaches in SFT ensure that each token in the attention calculation phase only focuses on tokens within its own short sequence, without providing additional learning signals for the preceding context. To address these challenges, we introduce Threshold Filtering Packing (TFP), a method that selects samples with related context while maintaining sufficient diversity within the same pack. Our experiments show that TFP offers a simple-to-implement and scalable approach that significantly enhances SFT performance, with observed improvements of up to 7% on GSM8K, 4% on HumanEval. Furthermore, results from bias benchmark datasets highlight TFP’s promising performance in improving fairness while also boosting prediction accuracy by 15%.

## introduction in supervised fine-tuning (sft) for large language models (llms), sequence lengths can vary substantially, requiring the wrapping of data into tensors to apply matrix operations optimized for cuda and gpus @xcite . as illustrated in figure 1 (a), vanilla fine-tuning pad shorter sequences with special tokens "[pad]" up to the maximum sequence length. while this ensures uniformity, it introduces inefficiencies by including irrelevant padding tokens in the computation, wasting gpu resources and dilutes the model's learning signal @xcite . to address this issue, packing sequences has become a common technique in autoregressive transformer models during training and inference to optimize context length and reduce padding @xcite . this method involves randomly selecting and concatenating data of varying lengths until reaching the designed maximum length. recent studies suggest that packed data, when batched and processed on multi-gpus, effectively minimize idle time within each batch @xcite . however, randomly concatenating these data samples (figure 1(b) ), can result in sequence cross-contamination @xcite . crosscontamination occurs when predictions for one sequence are influenced by an unrelated sequence, complicating accurate predictions, especially when the subjects differ. for instance, imagine a model tasked with generating a multiplication table, followed by a prompt to list useful expressions in french. if these two sequences are concatenated without proper separation, the model might mix the tasks, producing results like "3 x 2 = bonjour." this leads to incorrect and confusing outputs, where instructions and contexts become inappropriately blended. moreover, current sft pipelines @xcite cause previous samples to provide no signal for predicting the next sample, thereby reducing learning efficiency and negatively impacting the few-shot performance of llms. to address these challenges, in this work, we present threshold filtering packing (tfp), a new packing approach that packs sequences of related yet diverse samples, encouraging context richness and reasoning across sample boundaries. specifically, we employ a greedy algorithm inspired by the traveling salesman problem (tsp) @xcite to efficiently map out a path for segmentation into multiple packs. tfp further refine these packs by ensuring that overly similar samples are not grouped together. setting the threshold is crucial in this context as it allows us to strike a balance between similarity and diversity within each pack, preventing homogeneity and ensuring instruction1 output1 instruction3 output3 instruction2 output2 instruction4 output4 embedding2 8.4 embedding4 8.6 20.5 embedding1 20.4 embedding3 2.1 embedding1 8.4 embedding2 embedding3 15.7 embedding4 tsp 2.1 embedding1 8.4 embedding2 embedding3 15.7 embedding4 tfp 20.5 embedding1 embedding2 20.4 embedding3 8.4 embedding4 random ... ... ... 8.6 instruction1 output1 instruction3 output3 instruction2 output2 instruction4 output4 [pad] ... instruction1 output1 instruction3 output2 instruction2 output3 instruction4 output4 [pad] ... instruction1 output1 [pad] ... [pad] [pad] [pad] [pad] [pad] [pad] [pad] [pad] [pad] random packing: vanilla ft: threshold filtering packing: different from vanilla ft (a), which uses "[pad]" tokens up to the maximum length, and random packing (b), which places randomly shuffled samples in the same pack and may lead to cross-contamination, tfp (d) places related samples in the same pack while applying a threshold on tsp (c) to ensure that these samples are sufficiently distinct. this approach allows models to learn across diverse contexts and prevent overly similar samples from being grouped together. the robustness of the generated packs. as shown in figure 1 (c), each sample is first converted to an embedding and then represented as a node in the graph. as shown in figure 1 (d), tfp employs threshold filtering to ensure each sample is distinct enough from recent ones, preventing the model from merely replicating previous outputs. this method results in packs that provide useful context while avoiding cross-contamination from unrelated texts. for experiments, we fine-tune various llms on standard instruction fine-tuning datasets and conduct bias-related experiments to evaluate the potential effect of packing on fairness. additionally, we assess the impact of tfp on computational efficiency for sft. our findings indicate that tfp demonstrates superior performance across various models. the bias-related experiments show that tfp offers a flexible operational space, allowing for adjustments in the ratio of sensitive attributes (e.g., race) within packs to effectively manage bias. in contrast, tfp avoids the need for masking to prevent cross-contamination. instead, it forms data packs that provide relevant context without additional masking, streamlining implementation and reducing memory overhead. ## threshold filtering packing the standard practice in packing is to form a pack by concatenating random samples until reaching the maximum context length @xcite . however, randomly concatenated packs do not provide additional learning signals and can lead to cross-contamination of sequences, compared to training on individual samples. in contrast, tfp generates more coherent packs by concatenating related and useful samples together, improving sft performance and computational efficiency. ## experiment we evaluate the proposed approach from three aspects: (1) performance comparison: we compare tfp with various llms fine-tuned on common instruction fine-tuning datasets under both zero-shot and few-shot settings. (2) bias and fairness: inspired by previous research @xcite that studies the impact of the ratio of different demographic groups in in-context learning (icl) on llm fairness, we investigate how adjusting the ratio in each pack during sft can influence the bias and fairness of llms. (3) efficiency: we study how various sft methods influence computational efficiency on different gpu setups. ## conclusion we present tfp, a novel and scalable method for packing samples during sft, enabling language models to learn from relevant context and adapt effectively to few-shot evaluation. tfp is simple-toimplement and integrates with existing sft framework by adjusting the sample sequence. our evaluation shows tfp significantly boosts sft performance on both text and tabular data, especially in few-shot tasks. it also improves fairness by introducing similar texts with varying sensitive attributes, helping reduce biases in llms without compromising accuracy.