C hu L o: Chunk-Level Key Information Representation for Long Document Understanding (2025)

Abstract: Transformer-based models have achieved remarkable success in various Natural Language Processing (NLP) tasks, yet their ability to handle long documents is constrained by computational limitations. Traditional approaches, such as truncating inputs, sparse self-attention, and chunking, attempt to mitigate these issues, but they often lead to information loss and hinder the model’s ability to capture long-range dependencies. In this paper, we introduce ChuLo, a novel chunk representation method for long document understanding that addresses these limitations. Our ChuLo groups input tokens using unsupervised keyphrase extraction, emphasizing semantically important keyphrase based chunks to retain core document content while reducing input length. This approach minimizes information loss and improves the efficiency of Transformer-based models. Preserving all tokens in long document understanding, especially token classification tasks, is important to ensure that fine-grained annotations, which depend on the entire sequence context, are not lost. We evaluate our method on multiple long document classification tasks and long document token classification tasks, demonstrating its effectiveness through comprehensive qualitative and quantitative analysis.

## introduction transformer-based models @xcite , including llms @xcite @xcite @xcite @xcite , have achieved remarkable success across a wide range of natural language processing (nlp) tasks, including machine translation, text summarization, text generation, and text classification. a key fac-tor behind their success is the self-attention mechanism, which allows the model to capture longrange dependencies by computing the similarity between any two tokens and aggregating information accordingly. however, this mechanism incurs a quadratic computational cost in terms of both time and space, relative to input length. this computational burden makes it difficult for transformerbased models to scale to long documents, limiting their application to real-world data with unrestricted document lengths. to address this challenge, several approaches have been proposed for applying transformer-based models to long documents while managing computational resources. one of them is truncating, where the model discards content exceeding a predefined input length. for instance, @xcite processes up to 512 tokens, and llama @xcite handles up to 2048 tokens, with any additional content being ignored. another one is sparse self-attention, which reduces computational complexity by restricting each query token to attend only to a subset of key tokens @xcite @xcite @xcite . lastly, chunking divides long documents into smaller, manageable segments that are processed independently by the model @xcite . while these methods enable transformer-based models to process long documents, they have limitations. truncation risks discarding important information that falls beyond the maximum input length. although more efficient, sparse attention reduces each token's receptive field, leading to potential information loss from the neglected tokens. similarly, chunking breaks the input into isolated segments, which can disrupt long-range dependencies critical for a comprehensive understanding of the document. preserving all tokens is particularly important in tasks that require fine-grained tokenlevel understanding, such as token classification. in such tasks, dropping tokens can severely impact the accuracy of fine-grained annotations, which often depend on the full context of the document. therefore, there is a need for methods that can handle long documents efficiently while retaining all key information from the input. in this paper, we introduce chulo, a novel chunk-level key information representation method that addresses these challenges in long document classification and token classification. our method reduces input length while minimizing information loss by strategically grouping tokens using unsupervised keyphrase extraction. by identifying and emphasizing semantically important tokens, chulo ensures that each chunk retains the core content of the document. the resulting chunk representation is used for training transformer models, with more weight assigned to keyphrases to make them more salient in each chunk. we evaluate chulo on various long document classification tasks and long document token classification tasks, demonstrating its effectiveness through competitive results and thorough analysis. the key contributions of this paper are as follows: 1) novel chunk representation method: we introduce chulo, a chunk representation method for long document understanding that leverages unsupervised keyphrase extraction to prioritize semantically important information, effectively reducing input length while preserving core content. 2) enhanced document and token classification: our proposed method is designed to handle both document-level and token-level tasks, addressing the limitations of existing models in retaining fine-grained annotations and global context in long documents. 3) scalable and efficient solution: chulo offers a scalable and efficient approach for long document understanding, making it suitable for various nlp tasks where handling long-range dependencies and context preservation is critical. ## chulo we propose chulo, a novel chunk representation method that enhances long document understanding by reducing input length while preserving semantic content. unlike existing approaches like truncation and standard chunking, chulo minimizes information loss and maintains contextual dependencies. the method segments documents into non-overlapping chunks, integrates key semantic information using unsupervised keyphrase extraction, and assigns higher weights to keyphrase tokens in chunk representations. these enriched chunks train a transformer-based chunk attention module, enabling efficient processing of long documents while retaining global and local context. further details are provided in subsequent subsections, with the framework illustrated in figure 1 . ## experiments set-up we evaluate chulo on document and token classification tasks to highlight our motivation. while document classification primarily requires global contextual understanding, token classification tasks test the model's ability to retain and utilize detailed token-level information within long documents. we compare it with bert (kenton and toutanova, 2019) and bert variants @xcite , longformer @xcite , tobert @xcite , cogltx @xcite , chunkbert @xcite , and instructions with llms, gpt4o foot_1 and gemini1.5pro foot_2 . ## conclusion we introduced chulo, a novel chunk representation method that enhances the performance of transformer-based models on long document classification and token-level classification tasks. by utilizing unsupervised keyphrase extraction, chulo effectively reduces input length while preserving critical information, addressing the limitations of truncation and sparse attention. extensive experiments demonstrate that chulo outperforms existing methods by maintaining both global context and high accuracy, even for lengthy inputs. our research results highlight the effectiveness of chulo as a robust solution for long document understanding, enabling processing of complex texts in nlp applications.