Recurrent Attention Networks for Long-text Modeling (2023)

Abstract: Self-attention-based models have achieved remarkable progress in short-text mining. However, the quadratic computational complexities restrict their application in long text processing. Prior works have adopted the chunking strategy to divide long documents into chunks and stack a self-attention backbone with the recurrent structure to extract semantic representation. Such an approach disables parallelization of the attention mechanism, significantly increasing the training cost and raising hardware requirements. Revisiting the self-attention mechanism and the recurrent structure, this paper proposes a novel long-document encoding model, Recurrent Attention Network (RAN), to enable the recurrent operation of self-attention. Combining the advantages from both sides, the well-designed RAN is capable of extracting global semantics in both token-level and document-level representations, making it inherently compatible with both sequential and classification tasks, respectively. Furthermore, RAN is computationally scalable as it supports parallelization on long document processing. Extensive experiments demonstrate the long-text encoding ability of the proposed RAN model on both classification and sequential tasks, showing its potential for a wide range of applications.

## introduction recently, self-attention-based neural networks, such as transformer @xcite , gpt @xcite @xcite , and bert family @xcite @xcite , have demonstrated superior text encoding ability in many natural language processing (nlp) tasks with the help of large-scale pretraining. these models have set state-of-the-art benchmarks in classification tasks like text categorization @xcite and sentiment analysis @xcite @xcite , and sequential tasks like question answering @xcite and information extraction @xcite . the time and space complexities of self-attention computation are o(n 2 ) with respect to the sequence length, making it computationally expensive to encode long texts. therefore, bert models adopt an absolute positional encoding strategy to manage computational overhead. however, such a setting makes the bert models unable to handle texts longer than 512 tokens, restricting their application in realistic scenarios like processing user comments, news articles, scientific reports, and legal documents with arbitrary lengths. current works focus on two solutions to enable self-attention-based models for handling longer texts. the first solution reduces the computing complexity of self-attention from quadratic to linear by approximating its softmax operation @xcite @xcite . these models can handle relatively long texts within the hardware capacity but also suffer from a performance drop @xcite . another solution is to divide the long document into chunks shorter than 512 tokens so that pretrained bert models can be applied @xcite . however, as the chunks are individually encoded, the resulted representations do not contain the crucial contextual information for sequential tasks. while a special recurrent mechanism can handle sequential tasks @xcite , it cannot produce a document-level representation for classification, limiting their generality as none works for both classification and sequential tasks. additionally, introducing recurrent modules disables the parallel computing feature, leading to unscalable implementation. to address the aforementioned issues, this paper proposes the recurrent attention network (ran) foot_0 , a novel model architecture supporting recurrent self-attention operation over long sequences, enabling global dependency extraction and long-term memory. ran iterates through the sequence by non-overlapping windows. unlike token-level recurrent architectures such as lstm @xcite and gru @xcite , ran applies positional multi-head selfattention (pmhsa) on a window area to extract local dependency. to propagate the information forward, the ran model extracts the global perception cell (gpc) vector from the self-attention representation of the current window. the gpc vector is then concatenated with tokens in the next window as the input of the self-attention layer. the new gpc vector will be passed to the subsequent windows with residual connection to alleviate the gradient vanishing @xcite and updated in the same manner. figure 1 depicts the difference between the recurrent neural network (rnn) and our proposed ran. the function of the gpc vector is twofold. first, like the [cls] token in bert, the gpc vector is a window-level contextual representation. but unlike the [cls] token, the gpc vector is only applied to the self-attention layer, and no special token is inserted during text preprocessing. second, the gpc vector, resembling the state cell in a recurrent architecture, maintains a long-distance memory over the sequence. for each window, the attended gpc vector encodes an aggregated representation of all the previous windows, which enables the window-level self-attention to perceive global semantics. with the help of a well-designed memory review mechanism, the gpc vector from the last window can be used as a document-level representation and serve the classification tasks. meanwhile, the memory review mechanism enhances the token representations of ran in the sequence, encoding both contextual and global information, which can be leveraged for sequential tasks such as language modeling (lm) and named entity recognition (ner). we pretrain the ran model using a masked language modeling (mlm) objective from scratch, which outperforms other pretrained baselines in long document classification. the ran framework also supports auto-regressive lm and achieves the lowest perplexity score compared with state-of-theart language models on the wikitext-103 dataset. furthermore, we apply ran to different downstream tasks via finetuning and observe consistent improvements compared to baseline models. ran solely relies on self-attention, and no lstm-style gate is involved when propagating information via gpc vectors. therefore, ran is computationally efficient as it supports parallelized gpu computing. although the memory complexity is still quadratic, it is regarding the window size w rather than the whole text length l, where w ≪ l. nevertheless, the window size can be adjusted based on hardware availability to achieve a relatively larger batch size for better training. in summary, our contribution is to devise the ran model for long document processing. ran allows for parallelization on gpu and provides the interfaces for serving both classification and sequential tasks. with pretraining, ran can outperform the bert-based models in various tasks. ## related work this section reviews the relevant works focusing on sequence modeling in nlp, especially long document processing. rnns are widely used for sequential modeling by recursively updating a state cell to maintain a long-distance memory. traditional recurrent networks, such as lstm @xcite and gru @xcite , use the fully-connected layer as the basic encoding unit and apply the gate mechanism to update state memory. the recurrent operation is conducted on the token level, which is inefficient as such a framework cannot compute parallelly on gpu. besides, it might suffer from gradient vanishing for long sequences during the backpropagation phase @xcite . self-attention models are powerful in global representation learning. however, applying selfattention in long document processing is intractable due to the quadratic time and memory complexities. to address this issue, some works @xcite @xcite attempt to reduce the computing complexity of self-attention from quadratic to approximately linear complexity. @xcite propose a drop-in replacement of the softmax operation in self-attention with a sparse attention mechanism. similarly, choromanski et al. (2021) rely on prior knowledge like sparsity and low-rankness to efficiently estimate the full-rank attention. however, these approaches face a trade-off between efficiency and accuracy, as approximations may lead to a performance drop @xcite . other works leverage the power of full-rank selfattention as backbones, such as pretrained bert and roberta. these works cope with the tokenlength limitation with different strategies. @xcite propose cogltx framework to generate a brief summary of the document. the short summary is used for the classification task employing bert. however, it is inevitable to lose information in the length compression. @xcite segment the long text into smaller chunks so that bert can be then used. a recurrent layer is employed to obtain the document-level representation upon chunk-level representations. these models can be applied for the classification task but cannot handle sequential tasks because of losing crucial contextual and sequential information. @xcite adopt the chunking strategy and devise a specifically-designed gate mechanism to obtain token-level representations for sequential tasks. similarly, @xcite propose a transformer-based temporal latent bottleneck for image classification, reinforcement learning, and text classification, in which temporal states are updated using a recurrent function across chunks. in each transformer block, temporal states update the chunk-level representation by cross-attention layers interleaved with self-attention layers. in general, these models with bert backbones cannot simultaneously handle classification and sequential tasks for long documents. meanwhile, the rnn-style gate architecture does not support parallel computing, so the computing efficiency is also impaired. our proposed ran achieves recurrent operation of the self-attention model and hence supports par-allelization. moreover, like the traditional rnn architecture, ran can produce both token-level and document-level representations, which can be leveraged for both sequential and classification tasks. ## conclusion & future work this paper has presented a novel ran architecture for long-text modeling that combines the advantages of both recurrent and self-attention networks. the use of a positional multi-head attention mechanism and gpc vector enhances the model's performance by capturing both local and global dependencies in the input sequence. our ablation study also highlights the critical role of residual connection and memory review mechanisms in preserving long-distance memory. with the well-designed recurrent self-attention mechanism, ran's training can be accelerated by parallel computing on a gpu, making it highly efficient and scalable. we have conducted extensive experiments on tc, ner, and lm tasks. the extensive experiments demonstrate the effectiveness of the proposed ran model on both classification and sequential tasks. the flexibility and scalability of our proposed ran make it a promising choice for future research, with broad potential applications in translation, summarization, conversation generation, and large language models. additionally, we plan to extend the ran to tasks involving multi-modality input and output like audio and video, to exploit ran's long sequence handling capacity in different fields.