Long Context is Not Long at All: A Prospector of Long-Dependency Data for Large Language Models (2024)

Abstract: Long-context modeling capabilities are important for large language models (LLMs) in various applications. However, directly training LLMs with long context windows is insufficient to enhance this capability since some training samples do not exhibit strong semantic dependencies across long contexts.In this study, we propose a data mining framework ProLong that can assign each training sample with a long dependency score, which can be used to rank and filter samples that are more advantageous for enhancing long-context modeling abilities in LLM training. Specifically, we first use delta perplexity scores to measure the Dependency Strength between text segments in a given document. Then, we refine this metric based on the Dependency Distance of these segments to incorporate spatial relationships across long contexts. Final results are calibrated with a Dependency Specificity metric to prevent trivial dependencies introduced by repetitive patterns. Moreover, a random sampling approach is proposed to optimize the computational efficiency of ProLong. Comprehensive experiments on multiple benchmarks indicate that ProLong effectively identifies documents that carry long dependencies, and LLMs trained on these documents exhibit significantly enhanced long-context modeling capabilities.

## introduction large language models (llms) are widely used in many natural language processing (nlp) tasks @xcite . these tasks often require dealing with long text inputs @xcite , such as lengthy documents @xcite , long conversation histories in chatbots @xcite or large codebases @xcite . therefore, enhancing figure 1 : samples that carry longer dependencies better enhance llms' long-context modeling capabilities, even with a fixed training context window of 32k. llms to model long-context inputs is a prominent desiderata. there are primarily two categories of approaches to expand the context window of an llm. the first category fine-tunes llms with longer context windows @xcite , while the second category adjusts the llm's positional encoding or attention mechanism to accommodate larger position indices without further re-training @xcite . however, non-training methods often produce results inferior to those of finetuned llms @xcite , which model long-context inputs more effectively and generally achieve lower perplexity scores. although reported to be feasible, simply finetuning llms with naively sampled long corpora does not ensure improved long context modeling capabilities @xcite . some of these finetuned llms may still struggle to effectively process and utilize information from long input contexts even if they obtain a decently low perplexity score @xcite . this can lead to low performance in various downstream applications, even in some basic synthetic retrieval tasks @xcite . nevertheless, few approaches try to tackle this long-context modeling issue from a data-centric perspective. as revealed by @xcite , the quality of fine-tuning data plays a critical role in enhancing the long-context modeling capabilities of llms @xcite . besides, de vries (2023) also report that high-quality corpora significantly outperform other factors in boosting long-context performance. upon further exploration, we recognize that high-quality long-text data is characterized by the presence of long-range dependencies. the importance of encapsulating long-range dependencies in llms is also underscored by @xcite , which elucidates the benefits of integrating global dependencies into retrieval-augmented language models. however, such strong semantic dependencies are rare in typical training samples @xcite and diminish as the distance between segments increases @xcite . even with identical sequence lengths, different samples may exhibit varying dependency density. specifically, certain long training samples often comprise concatenated short documents that are randomly selected and do not have any semantic dependencies. moreover, even for inherently long documents, like novels, most tokens depend only to a brief span of preceding context. this phenomenon can be simply concluded as "long context is not long at all", leading to challenges in model learning (figure 1 ). therefore, we argue that explicitly incorporating long-dependency data into the fine-tuning process can facilitate long-context modeling. in this paper, we propose a novel framework (called prolong) to mine long-dependency data. prolong assigns a long-dependency score to each document, which serves as an indicator of the dependency density across long contexts. documents with higher scores are deemed more advantageous for boosting long context modeling, therefore we can use these scores to rank and filter high quality corpus for llm fine-tuning. concretely, prolong first partitions each document into fixed-length segments and evaluates dependency relationships between each segment pair from three perspectives: (i) dependency strength quantifies the difference in perplexities of a given segment when conditioned with or without its preceding segments. this metric measures the contribution of the preceding segment to the current one; (ii) dependency distance measures the positional gap and spatial relationship between two text segments; and (iii) dependency specificity employs entropy to ensure a non-uniform distribution of dependency strengths across all preceding segments, mitigating trivial dependencies introduced by repetitive patterns. a dependency score is assigned to each segment pair by combining the above three perspectives, and a final long-dependency score for the entire document is computed by accumulating dependency scores of all segment pairs. further, we adopt various strategies to optimize the computational efficiency of prolong, including sampling among segments, evaluating perplexity scores with small models and curating test sets for rapid validation. experiments on multiple benchmarks indicate that prolong effectively identifies documents that carry long dependencies and llms trained on these documents exhibit significantly enhanced long-context modeling capabilities. our contributions are summarized as follows: ## prolong framework figure 2 presents the overall framework of pro-long, which assigns a long dependency score (lds) for a given data sample s (s could be an intact document or a concatenated training sample). specifically, we first divide s into n segments of equal length c 1 , ..., c n . then for each segment pair (c j , c i ), @xmath0, three scores are calculated: 1. the dependency strength (dst) score evaluates how much semantic dependencies are there between c i and c j ; 2. the dependency distance (ddi) score demonstrates the distance between c i and c j ; 3. the dependency specificity (dsp) score penalizes the erroneous gains introduced by similar patterns in each segments. the final lds for s is obtained by merging the above three sets of scores. ## analysis 6.1 balance between accuracy and efficiency initial lds requires n 2 times precise calculations. in practice, sampling t ≪ n 2 times and using a small model for perplexity calculations provides a more efficient, albeit approximate, solution. in this section, we explore the balance between accuwe calculate the accuracy of prolong in the test set to retrieve long-context documents and the number of documents processed per second as a measure of speed. the experimental results are as shown in ## conclusion in this study, we propose prolong, a novel and efficient approach for filtering long-dependency data in language model extension training. benefiting from prolong, we demonstrate that longdependency data is the key to enhancing longcontext modeling capability. moreover, low-quality short-dependency data may even impair longcontext modeling capability. extensive experiments on multiple long-context benchmark datasets demonstrate the effectiveness of our prolong in enhancing the long-context modeling capability of large language models. we hope prolong can inspire researchers to spend more effort on data of more intrinsic long-dependency rather than long surface form.