Improving Continual Pre-training Through Seamless Data Packing (2025)

Abstract: Continual pre-training has demonstrated significant potential in enhancing model performance, particularly in domain-specific scenarios. The most common approach for packing data before continual pre-training involves concatenating input texts and splitting them into fixed-length sequences. While straightforward and efficient, this method often leads to excessive truncation and context discontinuity, which can hinder model performance. To address these issues, we explore the potential of data engineering to enhance continual pre-training, particularly its impact on model performance and efficiency. We propose Seamless Packing (SP), a novel data packing strategy aimed at preserving contextual information and enhancing model performance. Our approach employs a sliding window technique in the first stage that synchronizes overlapping tokens across consecutive sequences, ensuring better continuity and contextual coherence. In the second stage, we adopt a First-Fit-Decreasing algorithm to pack shorter texts into bins slightly larger than the target sequence length, thereby minimizing padding and truncation. Empirical evaluations across various model architectures and corpus domains demonstrate the effectiveness of our method, outperforming baselines in 99% of all settings. Code is available at https://github.com/Infernus-WIND/Seamless-Packing.

## introduction large language models (llms) have demonstrated remarkable capabilities in various natural language understanding and generation tasks @xcite @xcite . these models are usually pre-trained on extensive corpora spanning diverse domains @xcite , enabling them to perform a wide range of applications. recent studies have shown that continual pre-training is an effective strategy for adapting llms to specific domains or tasks @xcite @xcite , bridging the gap between general-purpose pre-training and task-specific finetuning @xcite . continual pre-training shares many similarities with standard pre-training, particularly in data packing methods, which involve splitting training data into fixed length sequences to facilitate parallelized training. however, while significant attention has been given to domain-specific data selection, sampling strategies, and mitigating catastrophic forgetting in continual pre-training @xcite @xcite , relatively little research has focused on the effective organization of training data within batches. currently, the most widely adopted method for data packing is straightforward: concatenating all texts into a single stream and segmenting it into fixed-length sequences based on the model's input size @xcite . while efficient and widely used, this approach has a notable drawback-arbitrary truncation of sequences within one piece of text often disrupts contextual continuity, resulting in incomplete sequences. such disruptions can degrade the model's ability to understand and generate coherent text, potentially leading to performance issues such as hallucination in downstream tasks @xcite . some existing data packing methods instead rely on padding to ensure uniform sequence lengths. however, padding introduces its own inefficiencies-it occupies valuable input space without any meaningful information, reducing the proportion of real data processed per step. this not only limits training efficiency but may also weaken the model's ability to leverage longer-range dependencies when a substantial portion of input sequences consists of meaningless tokens. to address these challenges, we frame the data packing process as an optimization problem aimed at maximizing the contextual continuity within sequences while minimizing truncation and padding, subject to constraints on the pre-defined sequence length and maximum overlapping contexts. we propose seamless packing (sp) as a solution. to optimize sequence utilization, our method follows a two-stage process with a sequential order. the first stage prioritizes long texts that meet specific criteria, dynamically generating overlapping sequences while maintaining a predefined maximum repetition ratio to avoid excessive redundancy. in conventional data packing, the final portion of a long text often fails to fill a complete sequence, requiring concatenation with subsequent texts. this results in fragmented chunks with insufficient context and additional truncation of the following text. by contrast, the first stage of our method maximizes contextual continuity and minimizes truncation by avoiding such fragmentation, enabling a seamless allocation of sequences, where a long text can fully occupy multiple consecutive sequences without requiring fragments from other texts. in the second stage, those shorter texts left behind by first stage are packed using an approximation algorithm, first-fit-decreasing (ffd). the packing bins are assigned a capacity slightly larger than the sequence length to reduce padding, while any tokens exceeding the sequence length are discarded to maintain efficiency. this stage ensures a seamless arrangement of text by minimizing both padding and truncation. these two stages contribute to seamless text packing from different angles, preserving contextual continuity. by integrating them, our method effectively addresses the optimization problem, resulting in an effective and systematic solution for data packing in continual pretraining. furthermore, our method complements and can be integrated with existing approaches that focus on data sampling or mitigating catastrophic forgetting, as it targets a distinct yet important aspect of continual pre-training-optimizing data packing. to evaluate the effectiveness of our method, we conduct comprehensive experiments on a variety of models and domains. the results demonstrate that our method consistently outperforms conventional data packing methods, achieving superior performance and generalization across diverse settings. the main contributions are summarized as follows: ## method in this section, we introduce seamless packing in detail. as illustrated in figure 1 , our method consists of two stages: sliding window and packing with dropping. the design of the two stages is motivated by the need to address two crucial challenges in data packing for continual pre-training: preserving contextual continuity and minimizing truncation and padding, the necessity of which has been highlighted in previous section. in the first stage, long texts that meet certain criteria are processed using a dynamic sliding window technique, effectively leveraging contextual continuity and minimizing truncation. the second stage processed the remaining short texts, reducing both truncation and padding. both stages are carefully designed to ensure the algorithm is seamless and improve the overall performance of llms in continual pretraining scenarios. ## experiments we conduct extensive experiments to assess the effectiveness of seamless packing across diverse domains and models. unless otherwise specified, models are independently pre-trained on domainspecific corpora. additionally, we design ablation studies to validate the contributions of both stages in our method. through comprehensive evaluations, we analyze the impact of hyperparameters, validate the theoretical analysis and further investigate key aspects of our method. the results demonstrate that our method consistently outperforms traditional approaches across all tasks, achieving a fourfold improvement over baseline methods (0.96% vs. 0.24%). all fine-tuning results are averaged over five runs (seeds 42-46). ## conclusion in this study, we introduce seamless packing, an innovative methodology for data packing in continual pre-training scenarios. we provide a rigorous theoretical analysis of our method to demonstrate its effect range. through a comprehensive series of experiments, we have systematically validated the effectiveness of our proposed approach. the empirical results reveal that our sliding window technique significantly enhances context preservation, while the packing with dropping strategy demonstrates remarkable efficacy in processing short text segments. these findings underscore the important role of data engineering in the model training paradigm, particularly highlighting that suboptimal data packing methods can substantially limit the utility of even high-quality datasets. we believe that our methodological contributions and empirical insights not only advance the current understanding of data packing in continual learning but also provide valuable perspectives for future research in this domain.