Analysing The Impact of Sequence Composition on Language Model Pre-Training (2024)

Abstract: Most language model pre-training frameworks concatenate multiple documents into fixed-length sequences and use causal masking to compute the likelihood of each token given its context; this strategy is widely adopted due to its simplicity and efficiency. However, to this day, the influence of the pre-training sequence composition strategy on the generalisation properties of the model remains under-explored.In this work, we find that applying causal masking can lead to the inclusion of distracting information from previous documents during pre-training, which negatively impacts the performance of the models on language modelling and downstream tasks. In intra-document causal masking , the likelihood of each token is only conditioned on the previous tokens in the same document, eliminating potential distracting information from previous documents and significantly improving performance. Furthermore, we find that concatenating related documents can reduce some potential distractions during pre-training, and our proposed efficient retrieval-based sequence construction method, Bm25Chunk, can improve in-context learning (+11.6%), knowledge memorisation (+9.8%), and context utilisation (+7.2%) abilities of language models without sacrificing efficiency.

## introduction large language models (llms) are pre-trained on large amounts of documents by optimising a language modelling objective and show an intriguing ability to solve a variety of downstream nlp tasks @xcite @xcite . previous works emphasise the importance of pre-training data quality (e.g., @xcite @xcite and diversity (e.g., @xcite @xcite https://github.com/yuzhaouoe/pretraining-data-packing @xcite to improve the generalisation properties of language models. however, the influence of the pre-training sequence composition strategy remains largely under-explored. for most decoder-only language model pretraining pipelines (e.g., @xcite @xcite @xcite @xcite , constructing a pre-training instance involves packing, which refers to the process of combining randomly sampled documents into a chunk that matches the size of the context window; and causal masking, which refers to predicting the next token conditioned on all previous tokens, including those from different documents in the chunk. an alternative to causal masking is intra-document causal masking, where the likelihood of each token is conditioned on the previous tokens from the same document; intra-document causal masking is not commonly used in existing open-source pretraining frameworks as it is argued to adversely impact pre-training efficiency @xcite . however, to the best of our knowledge, there is no systematic analysis in the literature on how causal masking affects the generalisation properties of models despite its role in improving efficiency. to analyse the impact of the packing and masking strategies on pre-training, we pre-train language models using intra-document causal masking (referred to as intradoc, section 2.2) and compare them with models pre-trained via causal masking with several packing strategies by varying the relatedness of the documents in the pre-training chunks. specifically, we analyse the results produced by a commonly used baseline method that randomly samples and packs documents (mixchunk); a method that samples and packs documents from the same source based on their meta-information (unichunk); and our proposed efficient retrievalbased packing method, which retrieves and packs related documents (bm25chunk, section 2.1). our experimental results indicate that using causal masking without considering the boundaries of documents can lead to the inclusion of distracting information from previous documents during pre-training (section 3 and section 5.1), negatively impacting the performance of the models in downstream tasks (section 4). we observe that intradocument causal masking, which eliminates the potential distractions from irrelevant documents during pre-training, can significantly improve the performance of the model while increasing its runtime (+4% in our implementation, see appendix a). we also find that improving the relatedness of the documents in pre-training chunks can reduce some potential distractions from previous documents (e.g., unichunk avoids packing documents from different distributions, such as code and news text), which can improve the performance of causal masking models on a wide array of downstream tasks. finally, we show that our proposed efficient retrieval-based packing method, bm25chunk, can improve a model's language modelling (+6.8%), in-context learning (+11.6%), knowledge memorisation (+9.8%), and context utilisation (+7.2%) abilities using causal masking and thus without sacrificing pre-training efficiency. our main contributions and findings can be summarised as follows: ## packing and masking strategies for pre-training sequence composition in this section, we formally introduce the pretraining data packing strategies, causal masking, and intra-document causal masking. ## experiments on downstream tasks in the following, we evaluate the in-context learning, knowledge memorisation, and context utilisation abilities of the models. ## conclusion in this work, we investigate the impact of pretraining sequence composition by pre-training models from scratch. first, we find causal masking can result in unrelated documents distracting language modelling pre-training and hurting the performance on downstream tasks; we show that intra-document 8 causal masking can significantly improve the performance while decreasing the pre-training efficiency. second, we find improving the relatedness of documents in pre-training chunks for causal masking pre-training can reduce some potential distractions in chunks; our proposed efficient retrievalbased packing method bm25chunk can significantly improve the performance of language models without reducing pre-training efficiency.