Hi-Transformer: Hierarchical Interactive Transformer for Efficient and Effective Long Document Modeling (2021)

Abstract: Transformer is important for text modeling. However, it has difficulty in handling long documents due to the quadratic complexity with input text length. In order to handle this problem, we propose a hierarchical interactive Transformer (Hi-Transformer) for efficient and effective long document modeling. Hi-Transformer models documents in a hierarchical way, i.e., first learns sentence representations and then learns document representations. It can effectively reduce the complexity and meanwhile capture global document context in the modeling of each sentence. More specifically, we first use a sentence Transformer to learn the representations of each sentence. Then we use a document Transformer to model the global document context from these sentence representations. Next, we use another sentence Transformer to enhance sentence modeling using the global document context. Finally, we use hierarchical pooling method to obtain document embedding. Extensive experiments on three benchmark datasets validate the efficiency and effectiveness of Hi-Transformer in long document modeling.

## introduction transformer @xcite is an effective architecture for text modeling, and has been an essential component in many state-of-the-art nlp models like bert @xcite @xcite . the standard transformer needs to compute a dense self-attention matrix based on the interactions between each pair of tokens in text, where the computational complexity is proportional to the square of text length @xcite . thus, it is difficult for transformer to model long documents efficiently @xcite . there are several methods to accelerate transformer for long document modeling @xcite @xcite . one direction is using transformer in a hierarchical manner to reduce sequence length, e.g., first learn sentence representations and then learn document representations from sentence representations @xcite . however, the modeling of sentences is agnostic to the global document context, which may be suboptimal because the local context within sentence is usually insufficient. another direction is using a sparse self-attention matrix instead of a dense one. for example, @xcite proposed to combine local self-attention with a dilated sliding window and sparse global attention. @xcite proposed to incorporate a random sparse attention mechanism to model the interactions between a random set of tokens. however, these methods cannot fully model the global context of document @xcite . in this paper, we propose a hierarchical interactive transformer (hi-transformer) foot_0 for efficient and effective long document modeling, which models documents in a hierarchical way to effectively reduce the complexity and at the same time can capture the global document context for sentence modeling. in hi-transformer, we first use a sentence transformer to learn the representation of each sentence within a document. next, we use a document transformer to model the global document context from these sentence representations. then, we use another sentence transformer to further improve the modeling of each sentence with the help of the global document context. finally, we use hierarchical pooling method to obtain the document representation. extensive experiments are conducted on three benchmark datasets. the results show that hi-transformer is both efficient and effective in long document modeling. shop the … without [cls] the royals … is [cls] prince philip … ) [cls] sentence 1 sentence 2 sentence m document 𝒆𝒆 1,1 𝒆𝒆 1,2 𝒆𝒆 1,𝐾𝐾 𝒆𝒆 𝑠𝑠 𝒉𝒉 1,1 𝒉𝒉 1,2 𝒉𝒉 1,𝐾𝐾 𝒉𝒉 1 𝑠𝑠 … … … 𝒆𝒆 2,1 𝒆𝒆 2,2 𝒆𝒆 2,𝐾𝐾 𝒆𝒆 𝑠𝑠 𝒉𝒉 2,1 𝒉𝒉 2,2 𝒉𝒉 2,𝐾𝐾 𝒉𝒉 2 𝑠𝑠 … … … 𝒆𝒆 𝑀𝑀,1 𝒆𝒆 𝑀𝑀,2 𝒆𝒆 𝑀𝑀,𝐾𝐾 𝒆𝒆 𝑠𝑠 𝒉𝒉 𝑀𝑀,1 𝒉𝒉 𝑀𝑀,2 𝒉𝒉 𝑀𝑀,𝐾𝐾 𝒉𝒉 𝑀𝑀 𝑠𝑠 word & position embedding word & position embedding word & position embedding + 𝒑𝒑 1 + 𝒑𝒑 2 + 𝒑𝒑 𝑀𝑀 𝒓𝒓 1 𝑠𝑠 𝒓𝒓 2 𝑠𝑠 𝒓𝒓 𝑀𝑀 𝑠𝑠 𝒅𝒅 1,1 𝒅𝒅 1,2 𝒅𝒅 1,𝐾𝐾 𝒅𝒅 1 𝑠𝑠 𝒅𝒅 2,1 𝒅𝒅 2,2 𝒅𝒅 2,𝐾𝐾 𝒅𝒅 2 𝑠𝑠 𝒅𝒅 𝑀𝑀,1 𝒅𝒅 𝑀𝑀,2 𝒅𝒅 𝑀𝑀,𝐾𝐾 𝒅𝒅 𝑀𝑀 𝑠𝑠 pooling pooling pooling … 𝒔𝒔 1 𝒔𝒔 2 𝒔𝒔 𝑀𝑀 pooling n× position embedding document contextaware sentence representations 𝒅𝒅 document embedding global context-aware sentence embedding document transformer sentence transformer sentence transformer sentence transformer sentence transformer sentence transformer sentence transformer 2 hi-transformer in this section, we introduce our hierarchical interactive transformer (hi-transformer) approach for efficient and effective long document modeling. its framework is shown in fig. 1 . it uses a hierarchical architecture that first models the contexts within a sentence, next models the document contexts by capturing the interactions between sentences, then employs the global document contexts to enhance sentence modeling, and finally uses hierarchical pooling techniques to obtain document embeddings. in this way, the input sequence length of each transformer is much shorter than directly taking the word sequence in document as input, and the global contexts can be fully modeled. the details of hi-transformer are introduced as follows.