Convolutions and Self-Attention: R e-interpreting Relative Positions in Pre-trained Language Models (2021)

Abstract: In this paper, we detail the relationship between convolutions and self-attention in natural language tasks. We show that relative position embeddings in self-attention layers are equivalent to recently-proposed dynamic lightweight convolutions, and we consider multiple new ways of integrating convolutions into Transformer self-attention. Specifically, we propose composite attention, which unites previous relative position encoding methods under a convolutional framework. We conduct experiments by training BERT with composite attention, finding that convolutions consistently improve performance on multiple downstream tasks, replacing absolute position embeddings. To inform future work, we present results comparing lightweight convolutions, dynamic convolutions, and depthwise-separable convolutions in language model pre-training, considering multiple injection points for convolutions in self-attention layers.

## introduction in recent years, transformer-based language models have brought dramatic improvements on a wide range of natural language tasks @xcite . the central innovation of transformer architectures is the self-attention mechanism @xcite , which has grown beyond nlp, extending into domains ranging from computer vision @xcite and speech recognition @xcite to reinforcement learning @xcite . in computer vision, self-attention and convolutions have been combined to achieve competitive results for image classification @xcite . similarly, researchers in nlp have begun integrating convolutions into self-attention for natural language tasks. recent work has shown initial success adding convolutional modules to self-attention in pre-trained language models @xcite , or even replacing self-attention entirely with dynamic convolutions @xcite . these successes defy theoretical proofs showing that multi-headed self-attention with relative position embeddings is strictly more expressive than convolution @xcite . to identify why convolutions have been successful in nlp, we seek to isolate the differences between self-attention and convolution in the context of natural language. in this work, we formalize the relationship between self-attention and convolution in transformer encoders by generalizing relative position embeddings, and we identify the benefits of each approach for language model pre-training. we show that self-attention is a type of dynamic lightweight convolution, a data-dependent convolution that ties weights across input channels @xcite . notably, previous methods of encoding relative positions @xcite are direct implementations of lightweight convolutions. under our framework, the benefits of convolution come from an ability to capture local position information in sentences. then, we propose composite attention, which applies a lightweight convolution that combines previous relative position embedding methods. we find that composite attention sufficiently captures the information provided by many other convolutions. to validate our framework, we train bert models that integrate self-attention with multiple convolution types, evaluating our models on the glue benchmark @xcite . all of our convolutional variants outperform the default model, demonstrating the effectiveness of convolutions in enhancing self-attention for natural language tasks. our empirical results provide evidence for future research integrating convolutions and self-attention for nlp. attention weights α ij are analogous to convolution kernel weights β j-i . ## self-attention and lightweight convolutions first, we outline the relationship between selfattention and convolutions. specifically, we show that a self-attention operation can be viewed as a dynamic lightweight convolution, a depthwise convolution that ties weights along channels @xcite . we then isolate the differences between self-attention and lightweight convolutions, highlighting the benefits of each approach in language models. ## integrating lightweight convolutions previous work has sought to integrate local information into global self-attention. this can be achieved by restricting the range of self-attention to nearby tokens, or by incorporating relative position information into attention maps @xcite @xcite . notably, @xcite introduced relative position em-beddings, which inspired similar embeddings in models such as transformer-xl and xlnet @xcite . in this section, we show that several previous methods of encoding relative positions are direct implementations of lightweight convolutions. ## non-lightweight convolutions the previous section found that lightweight convolutions consistently improved pre-trained language model performance. next, we investigated whether the additional flexibility of non-lightweight convolutions could provide additional benefits. specifically, we considered convolutions that were fixed but non-lightweight. in other words, convolution weights were fixed regardless of the input query, but weights were not tied across channels, equivalent to a standard depthwise convolution. we only considered fixed depthwise convolutions because under existing frameworks, dynamic depthwise convolutions would introduce large numbers of parameters. to implement depthwise convolutions, we added a convolution term identical to the fixed lightweight convolution in equation 4, except that β j-i was figure 3 : learned convolution kernel weights β j-i,c (equation 7 ) for the depthwise convolution in the deepest attention layer. channels correspond to the 256 features in each token representation. channels are sorted such that kernels differentiating the previous and next token are grouped together. this is equivalent to adding a depthwise convolution of the token values to the standard selfattention output. ## convolutional queries, keys, and values improvements over the non-convolutional baselines indicate that convolutions are beneficial to language model pre-training, serving as replacements for absolute position embeddings. our previous experiments applied different types of convolutions to self-attention values. to take this result one step further, we replaced the linear query, key, and value projections themselves with convolutional layers. intuitively, applying convolutions before selfattention induces even more mixing of token representations. if convolutions are built into every query, key, and value, then it becomes impossible for a token i to pay attention to a single token j without also incorporating information about tokens surrounding token j. ## discussion our results demonstrate that convolutions provide consistent benefits to pre-trained language models. our proposed composite attention mechanism combines previous relative position embedding methods, showing that convolutions can effectively compensate for the lack of local position information in transformer models. ## conclusion in this work, we formalized the relationship between self-attention and convolution. we proposed composite attention, which combines self-attention with lightweight convolution, uniting previous approaches to relative positions. our formulation and empirical results demonstrate that convolutions can improve self-attention by providing local position information in sentences, capable of replacing absolute position embeddings entirely. our findings provide a solid foundation from which to study convolutions and self-attention in language tasks. the spatially-oriented nature of convolutional neural networks translates directly into positional information in language. as vision and language researchers strive towards common deep learning architectures, it is important to recognize how architectures for vision tasks can be adapted to linguistic domains.