DA -Transformer: Distance-aware Transformer (2021)

Abstract: Transformer has achieved great success in the NLP field by composing various advanced models like BERT and GPT. However, Transformer and its existing variants may not be optimal in capturing token distances because the position or distance embeddings used by these methods usually cannot keep the precise information of real distances, which may not be beneficial for modeling the orders and relations of contexts. In this paper, we propose DA-Transformer, which is a distance-aware Transformer that can exploit the real distance. We propose to incorporate the real distances between tokens to re-scale the raw self-attention weights, which are computed by the relevance between attention query and key. Concretely, in different self-attention heads the relative distance between each pair of tokens is weighted by different learnable parameters, which control the different preferences on long- or short-term information of these heads. Since the raw weighted real distances may not be optimal for adjusting self-attention weights, we propose a learnable sigmoid function to map them into re-scaled coefficients that have proper ranges. We first clip the raw self-attention weights via the ReLU function to keep non-negativity and introduce sparsity, and then multiply them with the re-scaled coefficients to encode real distance information into self-attention. Extensive experiments on five benchmark datasets show that DA-Transformer can effectively improve the performance of many tasks and outperform the vanilla Transformer and its several variants.

## introduction transformer @xcite has achieved huge success in the nlp field in recent years @xcite . it serves as the basic architecture of various state-of-the-art models like bert @xcite and gpt @xcite , and boosts the performance of many tasks like text generation @xcite , machine translation @xcite , and reading comprehension @xcite . thus, the improvement on the transformer architecture would be beneficial for many nlp-related fields @xcite . a core component of transformer is multi-head self-attention, which is responsible for modeling the relations between contexts @xcite . however, self-attention is position-agnostic since it does not distinguish the orders of inputs. thus, in the vanilla transformer, position encoding is applied to the input to help transformer capture position information. however, in contrast to recurrent and convolutional neural networks, it is difficult for vanilla transformers to be aware of the token distances @xcite , which are usually important cues for context modeling. thus, several works explored to incorporate token distance information into transformer. for example, @xcite proposed to combine the embeddings of relative positions with attention key and value in the self-attention network. they restricted the maximum relative distance to only keep the precise relative position information within a certain distance. @xcite proposed a variant of self-attention network for named entity recognition, which incorporates sinusoidal embeddings of relative position to compute attention weights in a direction-and distance-aware way. however, the distance or relative position embeddings used by these methods usually cannot keep the precise information of the real distance, which may not be beneficial for the transformer to capture word orders and the context relations. in this paper, we propose a distance-aware transformer (da-transformer), which can explicitly exploit real token distance information to enhance context modeling by leveraging the relative distances between different tokens to re-scale the raw attention weights before softmax normalization. more specifically, since global and local context modeling usually have different distance preferences, we propose to learn a different parameter in different attention heads to weight the token distances, which control the preferences of attention heads on long or short distances. in addition, since the weighted distances may not have been restricted to a proper range, we propose a learnable sigmoid function to map the weighted distances into rescaled coefficients. they are further multiplied with the raw attention weights that are clipped by the relu function for keeping the non-negativity and introducing sparsity. we conduct extensive experiments on five benchmark datasets for different tasks, and the results demonstrate that our approach can effectively enhance the performance of transformer and outperform its several variants with distance modeling. the main contributions of this paper include: ## da-transformer in this section, we introduce our proposed distanceaware transformer (da-transformer) approach, which can effectively exploit real token distance information to enhance context modeling. it uses a learnable parameter to weight the real distances between tokens in each attention head, and uses a learnable sigmoid function to map the weighted distances into re-scaled coefficients with proper ranges, which are further used to adjust the raw attention weights before softmax normalization. the details of da-transformer are introduced in the following sections. ## conclusion in this paper, we propose a distance-aware transformer, which can leverage the real distance between contexts to adjust the self-attention weights for better context modeling. we propose to first use different learnable parameters in different attention heads to weight the real relative distance between tokens. then, we propose a learnable sigmoid function to map the weighted distances into re-scaled coefficients with proper ranges. they are further multiplied with the raw attention weights that are activated by the relu function to keep non-negativity and produce sharper attention. extensive experiments on five benchmark datasets show that our approach can effectively improve the performance of transformer by introducing real distance information to facilitate context modeling.