Graph-tree Fusion Model with Bidirectional Information Propagation for Long Document Classification (2024)

Abstract: Long document classification presents challenges in capturing both local and global dependencies due to their extensive content and complex structure. Existing methods often struggle with token limits and fail to adequately model hierarchical relationships within documents. To address these constraints, we propose a novel model leveraging a graph-tree structure. Our approach integrates syntax trees for sentence encodings and document graphs for document encodings, which capture fine-grained syntactic relationships and broader document contexts, respectively. We use Tree Transformers to generate sentence encodings, while a graph attention network models inter- and intra-sentence dependencies. During training, we implement bidirectional information propagation from word-to-sentence-to-document and vice versa, which enriches the contextual representation. Our proposed method enables a comprehensive understanding of content at all hierarchical levels and effectively handles arbitrarily long contexts without token limit constraints. Experimental results demonstrate the effectiveness of our approach in all types of long document classification tasks.

## introduction long document understanding has garnered increasing attention in the field of natural language processing (nlp) due to its wide range of applications across various domains, including legal document analysis, scientific literature categorization, and clinical text mining. accurate understanding of long documents is essential for tasks such as information retrieval, content summarization, and decision-making support systems. modern deep learning models for semantic analysis achieve impressive results by training on large datasets, which enables them to generate highly accurate predictions on unseen content @xcite . how-✉ corresponding author. ever, their ability to capture relationships between words and sentences relies on increasingly complex statistical operations as the text sequence lengthens @xcite . consequently, many existing methods become impractical for real-world applications, which makes processing long documents a challenging task. one of the primary challenges of long document classification is managing the large volume of information. unlike short texts, long documents contain extensive content that often spans multiple topics, making it difficult to capture the overall context of the document effectively. transformer-based models @xcite , such as bert @xcite , roberta @xcite , gpt-3 @xcite and @xcite , have gained popularity for nlp tasks due to their ability to capture (relatively) longrange dependencies and contextual relationships. however, in long document classification, transformer models face scalability issues due to their quadratic time complexity with respect to the input length. processing long documents with transformers can be computationally expensive and memoryintensive, often requiring substantial hardware resources. current methods for handling lengthy documents include truncating texts to a predefined length or modifying the attention mechanism. truncating to the first 512 tokens is straightforward but may cause significant information loss. sparse attention models like longformer @xcite and big bird @xcite ) reduce computational load by focusing on a subset of tokens, but they do not fully capture comprehensive context and dependencies in long texts. another significant challenge in long document classification is capturing contextual dependencies and the hierarchical structure. long documents have dependencies at word, sentence, and document levels, which are crucial for understanding content and leveraging both local and global con-texts. this structure is important for understanding the overall context and meaning. treating text as a flat sequence of tokens can cause models to miss these important hierarchical relationships. current approaches to address these challenges involve hierarchical models. for instance, hierarchical attention network (han) @xcite and hierarchical attention transformers (hat) @xcite , aim to capture both wordlevel and sentence-level representations before aggregating them into document-level embeddings. however, these models often fail to capture the intricate relationships between different parts of the document, such as the interplay between words, sentences, and overall document structure @xcite . additionally, hierarchical models may struggle with long-range dependencies @xcite , which misses the relationships between distant sections essential for understanding the overall context. to address the aforementioned challenges and constraints, we propose a novel model that leverages a graph-tree structure for arbitrarily long document classification. our approach fuses syntax trees for sentence encodings with document graphs for document encodings, which provides a comprehensive representation that captures both local and global dependencies, respectively. syntax trees (both dependency and constituency) represent the grammatical structure of sentences, which enhances sentence-level understanding. we use tree transformers @xcite to generate sentence encodings from these syntax trees. the document graph preserves hierarchical relationships within the document, which ensures that both local and global contexts are considered during the classification process. we apply the graph attention network (gat) @xcite on the constructed document graph to model the dependencies between the sentences. this graph structure effectively captures both inter-and intra-sentence dependencies. meanwhile, during training, we implement a bidirectional information propagation approach where information flows both from word-to-sentence-to-document and from documentto-sentence-to-word. this bidirectional flow enriches the contextual representation of the document, which allows for a more comprehensive understanding of the content at all hierarchical levels. by incorporating syntax trees and document graphs, we can encode text using different semantic units, such as sentence-level and document-level repre-sentations, based on their unique characteristics. this allows our model to handle arbitrarily long contexts without being constrained by token limits. to summarize, our main contributions are: ## method we propose a novel graph-tree fusion model, as shown in figure 1 that leverages a graph-tree structure for long document classification. our model uses multi-granularity document representations through tree transformers and graph attention networks. it fuses syntax trees for sentence encodings with document graphs for document encodings. additionally, during training, we implement a bidirectional information propagation approach, allowing information to flow both from word-to-sentence-to-document and from documentto-sentence-to-word. ## related work long documents present unique challenges. as the document length increases, maintaining context becomes increasingly difficult, which makes the task substantially more complex compared to short text classification @xcite . early methods relied on feature extraction techniques, where document length was not a significant issue. however, this changed with deep learning approaches using cnns and rnns that were implemented at different semantic levels, including character, word, and sentence levels @xcite . cnns often focus on local dependencies @xcite , while rnns @xcite are built for long-range dependencies. transformer-based models, such as bert @xcite and xlnet @xcite , have since come to dominate, but are limited by the number of tokens they can process, which poses a challenge for handling long documents. methods for modifying transformer architectures to handle long documents include recurrent transformers and sparse attention transformers @xcite . transformer-xl @xcite , a recurrent approach, introduced a segment-level recurrence mechanism and a positional encoding scheme, which enabled the model to capture longterm dependencies more effectively. another approach, ernie-doc @xcite , incorporated a continuous multi-segment attention mechanism and entity-aware pre-training to capture comprehensive contextual information across longer texts. alternatively, sparse transformers @xcite reduced the computational complexity of self-attention by selectively focusing on a subset of relevant tokens rather than all tokens in a sequence. reformer @xcite then improved the efficiency of transformers by using locality-sensitive hashing for sparse attention and reversible layers to reduce memory usage. besides the aforementioned approaches, methods such as longformer @xcite and big bird @xcite used a combination of local and global attention mechanisms to reduce computational complexity of standard self-attention. recently, @xcite used ssm to address the computational challenges (quadratic complexity in self-attention) caused by processing long sequences with traditional transformers. they introduced an ssm-pooler model, which incorporates a max pooling layer between each ssm block. this design allows the model to automatically extract important information from nearby inputs at each level and reduce the input length to half of that in the previous layer, which significantly accelerates the speed of both training and inference. they compared their model with self-attention-based models and achieved comparable performance while being, on average, 36% more efficient. ## conclusion in this paper, we address challenges of long document classification by leveraging a novel graph-tree structure. by integrating syntax trees for sentence encodings and document graphs for comprehensive document encodings, our approach captures both fine-grained syntactic relationships and broader contextual dependencies. using tree transformers and gat ensures accurate modeling of hierarchical relationships within documents. additionally, our bidirectional information propagation technique enhances the contextual representation, which enables a deeper understanding of content at all hierarchical levels. notably, our approach not only overcomes the limitations of token constraints but also improves the performance and accuracy of long document classification, making it highly suitable for long document understanding. potential extensions of this work could involve incorporating external knowledge through the integration of knowledge graphs. by linking document content with relevant external information, the model can further enhance its understanding and contextawareness and it would open new avenues for applications and improve the model's versatility across various domains.