L ong A ttn: Selecting Long-context Training Data via Token-level Attention (2025)

Abstract: With the development of large language models (LLMs), there has been an increasing need for significant advancements in handling long contexts. To enhance long-context capabilities, constructing high-quality training data with **long-range dependencies** is crucial. Existing methods to select long-context data often rely on sentence-level analysis,which can be greatly optimized in both performance and efficiency. In this paper, we propose a novel token-level framework, ​**LongAttn**​, which leverages the self-attention mechanism of LLMs to measure the **long-range dependencies** for the data. By calculating token-level dependency strength and distribution uniformity of token scores, LongAttn effectively quantifies ​**long-range dependencies**​, enabling more accurate and efficient data selection. We filter **LongABC-32K** from open-source long-context datasets (ArXiv, Book, and Code). Through our comprehensive experiments, LongAttn has demonstrated its excellent ​**effectiveness**​, ​**scalability**​, and ​**efficiency**​. We will release our code and the high-quality long-context dataset **LongABC-32K** in the future.

## introduction large language models (llms) have achieved impressive performance across a broad spectrum of traditional natural language processing tasks @xcite . to effectively address real-world applications, these models further require enhanced capabilities in handling longer contexts, particularly in key areas such as in-context learning @xcite , real-world questionanswering based on lengthy documents @xcite , long-context dialogue with historical context @xcite , and comprehensive document summarization @xcite . to enhance llms' long-context processing capabilities, data engineering remains fundamental. simple methods to construct long-context datasets are through naive methods like concatenating short texts or randomly sampling existing sources (e.g., commoncrawl, github). however, studies by de vries (2023) and @xcite emphasize that data obtained through such approaches fail to effectively improve long-context capabilities of llms because the data lack meaningful long-range dependencies. inspired by this, a line of studies focus on exploring the identification and selection of high-quality long-context with consideration of relations between text segments were proposed. prolong @xcite ) measures long-range dependencies between segments based on the relative perplexity and relative distance. @xcite develop a set of metrics including complexity, coherence, and cohesion based on various kinds of text segments (i.e., sliding windows, sentences, paragraphs) to measure the quality of long texts. however, these methods have two main drawbacks: (1) linguistic metrics do not fully align with the underlying mechanisms of llms, as they often fail to capture fine-grained token-level relationships. (2) they are computationally expensive and inefficient. for example, prolong reports that the speed of a 7b parameter model is roughly 1/16 of that of a 350m parameter model, making such methods challenging to scale for llms. attention mechanisms have been proven to effectively model context understanding @xcite . some studies focusing on attention mechanisms and positional encoding have shown that they can significantly improve a model's long-context ability @xcite . motivated by this, we propose to address the limitations of sentence-level selection methods leveraging the rich information provided in the attention mechanism. specifically, we propose longattn, a simple yet effective frame-the variance of the matrix distribution is used to measure the uniformity of token distribution (𝑫𝑼 𝑻 ). ## methodology as shown in figure 2 , our proposed method can be divided into three steps. firstly, we gather and preprocess the data to a predetermined length. subsequently, we employ the self-attention mechanism of a llm to compute the long-distance dependency score for each data instance. finally, we filter the data based on the score and utilize the refined dataset for continued pre-training of the model. ## experimental results we validate the effectiveness, scalability, and high efficiency of longattn through a series of datascale and lateral comprehensive experiments. ## analysis 6.1 ablation study to investigate the impact of the constraint factor α and the correction term du t on regulating lds t , we conduct ablation experiments on the d a3 and d a5 datasets using retrieval tasks. the default setting of the constraint factor α is 0.5. as shown in the ## conclusion in this paper, we introduce longattn, a framework evaluates long-range dependencies at the token level. longattn is effective as the self-attention mechanism captures relationships between all token contexts during inference. this approach to measuring long-range dependencies aligns better with the underlying operating principles of llms. we validate the effectiveness, scalability, and high efficiency of longattn through a series of comprehensive experiments. additionally, our research contributes to the previously limited study of highquality long-context training data. this finding suggests promising directions for future research, and we anticipate further advancements in this domain through subsequent investigations.