H i P ool: Modeling Long Documents Using Graph Neural Networks (2023)

Abstract: Encoding long sequences in Natural Language Processing (NLP) is a challenging problem. Though recent pretraining language models achieve satisfying performances in many NLP tasks, they are still restricted by a pre-defined maximum length, making them challenging to be extended to longer sequences. So some recent works utilize hierarchies to model long sequences. However, most of them apply sequential models for upper hierarchies, suffering from long dependency issues. In this paper, we alleviate these issues through a graph-based method. We first chunk the sequence with a fixed length to model the sentence-level information. We then leverage graphs to model intra- and cross-sentence correlations with a new attention mechanism. Additionally, due to limited standard benchmarks for long document classification (LDC), we propose a new challenging benchmark, totaling six datasets with up to 53k samples and 4034 average tokens’ length. Evaluation shows our model surpasses competitive baselines by 2.6% in F1 score, and 4.8% on the longest sequence dataset. Our method is shown to outperform hierarchical sequential models with better performance and scalability, especially for longer sequences.

## introduction transformer-based models like bert @xcite and roberta @xcite have achieved satisfying results in many natural language processing (nlp) tasks thanks to largescale pretraining @xcite . however, they usually have a fixed length limit, due to the quadratic complexity of the dense self-attention mechanism, making it challenging to encode long sequences. one way to solve this problem is to adapt transformers to accommodate longer inputs and optimize the attention from bert @xcite . bigbird @xcite applies sparse attention that combines random, global, and sliding window attention in a long sequence, reducing the quadratic dependency of full attention to linear. similarly, longformer @xcite applies an efficient self-attention with dilated windows that scale linearly to the window length. both models can take up to 4096 input tokens. though it is possible to train even larger models for longer sequences, they are restricted by a pre-defined maximum length with poor scalability. more importantly, they fail to capture high-level structures, such as relations among sentences or paragraphs, which are essential to improving nlp system performance @xcite . another way is to apply a hierarchical structure to process adjustable input lengths with chunking representations for scalability on long sequences. hi-transformer @xcite encodes both sentence-level and document-level representations using transformers. tobert @xcite applies a similar approach that stacks a sentence-level transformer over a pretrained bert model. while most of the existing work models upper-level hierarchy using sequential structures, such as multiple layers of lstms @xcite or transformers, this may still bring the long dependency issue when the sequence gets longer. to alleviate this, we investigate graph modeling as a novel hierarchy for upper levels. besides, we also consider inter-hierarchy relationships using a new attention mechanism. our key insight is to replace the sequence-based model with a hierarchical attentional graph for long documents. we first apply a basic pretrained language model, bert or roberta, to encode local representation on document chunks with a fixed length. the number of chunks could be extended for longer sequences for better scalability. different from other works, we apply a graph neural network (gnn) @xcite to model the upper-level hierarchy to aggregate local sentence in-formation. this is to alleviate the long dependency issue of the sequential model. moreover, within such a graph structure, we propose a new heterogeneous attention mechanism to consider intra-and cross-sentence-level correlations. our contributions are two-fold: 1) we propose hipool with multi-level hierarchies for long sequence tasks with a novel inter-hierarchy graph attention structure. such heterogeneous graph attention is shown to outperform hierarchical sequential models with better performance and scalability, especially for longer sequences; 2) we benchmark the ldc (long document classification) task with better scaled and length-extended datasets. evaluation shows that hipool surpasses competitive baselines by 2.6% in f1 score, and 4.8% on the longest sequence dataset. code is available at https: //github.com/irenezihuili/hipool . ## model we introduce the hipool (hierarchical pooling) model for long document classification, illustrated in fig. 1 . it consists of an overlapping sequence encoder, a hipool graph encoder, and a linear layer. overlapping sequence encoder. given the input document s, we first chunk the document into a number of shorter pieces with a fixed length l, and we set the overlapping window size to be l olp . overlapping encoding makes it possible for a chunk to carry information from its adjacent chunks but not isolated, differentiating our model from other hierarchical ones. then each chunk is encoded with a pretrained transformer model, i.e., bert or roberta; we choose the cls token representation as the input to our hipool layer: @xmath0. hipool graph encoder. we apply a graph neural network to encode incoming word-level information. such a model has shown its potential in some nlp tasks @xcite . we construct a graph, defined by g(v, e), where v is a set of nodes, and e is a set of node connections. there are two node types: n low-level nodes and m high-level nodes, and typically @xmath1. in our experiment, we set @xmath2, and @xmath3. the feedforward operation goes from low-to high-level nodes. in layer l, low-level nodes are inputs from the previous layer l -1, while high-level nodes at layer l are computed based on low-level ones. moreover, these high-level nodes will be the input to the next layer l + 1, becoming the low-level nodes in that layer. we consider x the low-level ## conclusion in this paper, we proposed a hierarchical framework for long document classification. the evaluation shows our model surpasses competitive baselines.