Adaptive Attention for Sparse-based Long-sequence Transformer (2023)

Abstract: Recently, Transformers have been widely used in various fields and have achieved remarkable results. But it is still difficult for Transformer-based models to process longer sequences because self-attention in them scales quadratically with the sequence length. Although some models attempt to use sparse attention to reduce computational complexity, hand-crafted attention patterns are unable to select useful tokens adaptively according to the context. Thus, in this paper, we propose a novel efficient Transformer model with adaptive attention, A2-Former, for long sequence modeling. It can select useful tokens automatically in sparse attention by learnable position vectors, which consist of meta position and offset position vectors. Because the learnable offset position is not an integer vector, we utilize the interpolation technique to gather corresponding vectors from the input embedding matrix by discrete indexes. Experiments on Long Range Arena (LRA), a systematic and unified benchmark with different tasks, show that our model has achieved further improvement in performance compared with other sparse-based Transformers.

## introduction transformer-based models @xcite have achieved state-of-the-art performance on a wide variety of natural language processing tasks @xcite @xcite . it is also gradually applied to other research fields such as speech and computer vision @xcite @xcite @xcite . although self-attention module, the core component in transformer, can capture global contexts from the whole sequence, the time and memory complexity are both quadratic to the sequence length. especially when facing longer sequences, transformer becomes more difficult to process them efficiently and effectively. recently, a wide spectrum of efficient transformers @xcite @xcite @xcite @xcite @xcite @xcite @xcite @xcite @xcite have been proposed to tackle the problem of efficiency, which can be roughly divided into the following directions: sparse attention, low-rank and kernel methods. because sparse-based attention is intuitive and interpretable in addition to efficiency, we focus on this method in this paper. it usually utilizes some strategies or patterns to limit the number of tokens involved in the attention calculation. different from traditional sparse transformer @xcite @xcite with different softmax and pattern-related quadratic computation, recent works mainly adopt sliding windows to achieve linear complexity. for example, longformer @xcite employs an attention pattern that combines local windowed attention with task-motivated global attention while also scaling linearly with the sequence length. bigbird @xcite incorporates random attention (queries attend to random keys) besides global tokens and local sliding windows. however, these hand-crafted attention patterns mentioned above are usually selected empirically or randomly. it is not an ideal solution for modeling long sequences. how to adaptively select useful tokens for sparse attention according to the context is still an important problem to be considered. to address these issues, we propose a 2 -former with adaptive attention to model longer sequences in this paper. it can select useful tokens automatically in sparse attention by learnable position vectors, which consist of meta position and offset position vectors. because each element in the learnable offset position vector is not an integer, we utilize linear interpolation to gather discrete vectors from original the input embedding matrix. position visualization further shows that traditional attention patterns are not enough to cover the valuable positions automatically selected by models. experiments on long range arena, a systematic and unified benchmark with different tasks, show that our model has achieved further improvement in performance compared with other sparse-based transformers. overall, the main contributions are as follows: ## related work recently, transformer @xcite and its variants @xcite @xcite have been widely used in natural language processing @xcite @xcite @xcite @xcite , computer vision @xcite @xcite , speech @xcite @xcite and other domains. to improve computational and memory efficiency, a dizzying number of efficient transformers @xcite @xcite @xcite @xcite @xcite @xcite @xcite @xcite @xcite have been proposed recently, which can be roughly divided into two directions: sparse attention, low-rank and kernel methods. sparse attention methods usually limit the field of view to fixed or random patterns. these patterns can also be used in combination. for example, sparse transformer @xcite combines stride and fixed factorized attention by assigning half of its heads to the pattern for reducing the complexity of a traditional transformer. longformer @xcite integrates a windowed localcontext self-attention and task-oriented global attention that encodes inductive bias about the corresponding task. bigbird @xcite incorporates random attention besides global attention and local window attention. random attention means that each query attends to a small number of random keys. however, it is still difficult for these hand-crafted, random or combined attention patterns to select valuable pairs in the sparse attention calculation. different from them, our proposed sparse attention mechanism can automatically and efficiently learn the position that should be selected and calculated. especially, our model is also different from traditional sparse transformer @xcite @xcite . they only focus on sparse softmax and its threshold and still require quadratic computation to determine the sparsity pattern. low-rank and kernel methods are the other solutions to improve the efficiency of transformer. low-rank methods usually assume a low-rank structure in the self-attention matrix. for example, linformer @xcite decomposes the original scaled dot-product attention into multiple smaller attentions through linear projections, such that the combination of these operations forms a low-rank factorization of the original attention. and kernel methods rewrite the self-attention mechanism through kernelization. for example, performer @xcite scales linearly rather than quadratically in the number of tokens in the sequence, which is characterized by subquadratic space complexity and does not incorporate any sparsity pattern priors. different from these mathematical and theoretical methods, our proposed method is still based on sparse attention but focuses more on how to find and learn attention patterns effectively and efficiently. ## conclusion in this paper, we propose a novel sparse-based transformer, a 2 -former, which replaces handcrafted attention patterns with learnable adaptive attention in sparse attention. we creatively adopt an interpolation technique to help the model gather discrete positions with continuous position vectors. by combining the meta position and generated offset position, the position of tokens can be selected dynamically according to the context. and position visualization further shows that traditional attention patterns are not enough to cover the useful positions automatically selected by models. experiments on lra show that our model has been significantly improved compared with the previous sparse transformers based on sliding windows.