Repulsive Attention: Rethinking Multi-head Attention as B ayesian Inference (2020)

Abstract: The neural attention mechanism plays an important role in many natural language processing applications. In particular, multi-head attention extends single-head attention by allowing a model to jointly attend information from different perspectives. However, without explicit constraining, multi-head attention may suffer from attention collapse, an issue that makes different heads extract similar attentive features, thus limiting the model’s representation power. In this paper, for the first time, we provide a novel understanding of multi-head attention from a Bayesian perspective. Based on the recently developed particle-optimization sampling techniques, we propose a non-parametric approach that explicitly improves the repulsiveness in multi-head attention and consequently strengthens model’s expressiveness. Remarkably, our Bayesian interpretation provides theoretical inspirations on the not-well-understood questions: why and how one uses multi-head attention. Extensive experiments on various attention models and applications demonstrate that the proposed repulsive attention can improve the learned feature diversity, leading to more informative representations with consistent performance improvement on multiple tasks.

## introduction multi-head attention @xcite is an effective module in deep neural networks, with impressive performance gains in many naturallanguage-processing (nlp) tasks. by extending a single head to multiple paralleled attention heads, the architecture is widely adopted to capture different attentive information and strengthen the expressive power of a model. @xcite applied the idea of multi-heads on self-attention and extract a 2-d matrix instead of a vector to represent different contexts of a sentence. the transformer @xcite and its variants such as bert @xcite are influential architectures solely based on multi-head attention, achieving state-of-the-art performance on plenty of nlp tasks. the key of multi-head attention is its ability to jointly attend to information from different representation subspaces at different positions, which results in multiple latent features depicting the input data from different perspectives. however, there are no explicit mechanisms guaranteeing this desired property, leading to potential attention redundancy or attention collapse, which has been observed in previous research @xcite . although there exist works by directly adding regularization on loss functions to encourage diversity in multi-head attention @xcite , the underlying working principle has not been well-validated, and performance improvement is limited. furthermore, an important problem on why and how multi-head attention improves over its single-head counterpart is poorly understood. in this paper, we provide a novel understanding of multi-head attention from a bayesian perspective by adapting the deterministic attention to a stochastic setting. the standard multi-head attention can be understood as a special case of our framework, where attention-parameter updates between heads are independent, instead of sharing a common prior distribution. based on our framework, attention repulsiveness could then be imposed by performing bayesian inference on attention parameters with the recently developed particle-optimization sampling methods @xcite , which has been shown to be effective in avoiding mode collapse. these methods treat each head as a particle/sample, which is then optimized to approximate a posterior distribution of an attention model. with it, multiple heads are enforced to move to modes in the parameter space to be far from each other, thus improving the repulsiveness in multi-head at-tention and enhancing model's expressiveness. our bayesian interpretation also provides a theoretical understanding of the reason and benefits of applying multi-head attention. experiments on various attention models demonstrate the effectiveness of our framework. our contributions are summarized as follow: ## a bayesian inference perspective of multi-head attention in this section, we interpret multi-head attention as bayesian inference of latent representation via particle-optimization sampling. we denote x and z as the input and output (latent representation) of the attention model, respectively. the single-head attention can be written as a deterministic mapping @xmath0; θ), with θ the parameter of the mapping. standard multi-head attention defines multiple parallel attention mappings, each endowed with independent parameters. the attention features are finally aggregated via a function g(•) as next, we generalize (6) as a bayesian inference problem for the latent representation z. ## repulsive attention optimization the bayesian multi-head attention in (7) further inspires us to develop the repulsive attention. the idea is to learn to generate repulsive samples from the posterior p(θ|d). we propose to adopt the particle-optimization sampling methods, which could explicitly encourage repulsiveness between samples. in our algorithm, the parameter of p(z|x; θ) for each head is considered as one particle. following the particle-optimization rules, m heads {θ i } m @xmath1. ## conclusion we propose a principled way of understanding multi-head attention from a bayesian perspective. we apply particle-optimization sampling to train repulsive multi-head attention with no additional trainable parameters nor explicit regularizers. our bayesian framework explains the long-standing question of why and how multi-head attention affects model performance. extensive experimental results on representative attention models demonstrate that our approach can significantly improve the diversity in multi-head attention, resulting in more expressiveness attention models with performance improvement on a wide range of tasks.