Guiding Attention for Self-Supervised Learning with Transformers (2020)

Abstract: In this paper, we propose a simple and effective technique to allow for efficient self-supervised learning with bi-directional Transformers. Our approach is motivated by recent studies demonstrating that self-attention patterns in trained models contain a majority of non-linguistic regularities. We propose a computationally efficient auxiliary loss function to guide attention heads to conform to such patterns. Our method is agnostic to the actual pre-training objective and results in faster convergence of models as well as better performance on downstream tasks compared to the baselines, achieving state of the art results in low-resource settings. Surprisingly, we also find that linguistic properties of attention heads are not necessarily correlated with language modeling performance.

## introduction recent advances in self-supervised pretraining @xcite @xcite have resulted in impressive downstream performance on several nlp tasks @xcite . however, this has led to the development of enormous models, which often require days of training on non-commodity hardware (e.g. tpus) @xcite . furthermore, studies have shown that it is quite challenging to successfully train these large transformer models @xcite , requiring complicated learning schemes and extensive hyperparameter tuning @xcite @xcite . despite these expensive training regimes, recent studies have found that once trained, these bi-directional language models exhibit simple patterns of self-attention without much linguistic backing (voita et al., 2019; raganato and tiedemann, 1 code: https://github.com/ameet-1997/attentionguidance attention guided model roberta-only mlm 0% 1% 100% figure 1 : attention patterns of our model (left) and the default roberta model (right) after 0% (top), 1% (middle) and 100% (bottom) of pre-training. inducing simple patterns (left) using an auxiliary loss leads to benefits in convergence speed, downstream performance, and robustness to hyperparameters. 2018). for example, 40% of heads in a pre-trained bert model @xcite ) simply pay attention to delimiters added by the tokenizer (e.g. in this paper, we propose an attention guidance (ag) mechanism for self-attention modules in transformer architectures to enable faster, more efficient, and robust self-supervised learning. our approach is simple and agnostic to the training objective. specifically, we introduce an auxiliary loss function to guide the self-attention heads in each layer towards a set of pre-determined patterns (e.g. figure 1 @xcite ). these patterns encourage the formation of both global (e.g. attend to through several experiments, we show that our approach enables training large transformer models considerably faster -for example, we can train a 16-layer roberta model with sota performance on a low-resource domain in just two days using four gpus, while excluding our loss leads to slow or no convergence. our method also achieves competitive performance with bert @xcite on three english natural language understanding tasks, and outperforms the baseline masked language modeling (mlm) models on eleven out of twelve settings considered. further, we also show that our initialization is agnostic to the training objective by demonstrating gains on the replaced token detection objective proposed by electra @xcite and on machine translation with transformers. finally, we provide an analysis of the attention heads learned using our method. surprisingly, contrary to recent studies @xcite , we find that it is possible to train models that perform well on language modeling without learning a single attention head that models coreferences. to summarize, our main contributions are: ## related work improving efficiency of lms the high computational costs of bert-style models have accelerated research on developing efficient contextual language models. @xcite used a gan-like setup to predict if each word in the input sequence is corrupted by a generator (another pre-trained lm). they show that their method is more sample efficient than the standard mlm objective. other studies have explicitly focused on making the selfattention modules more efficient. reformer @xcite and sparse transformer @xcite introduce locality-sensitive hashing and sparse factorizations to reduce the quadratic complexity of dot-product attention, while longformer @xcite uses local-windowed and task motivated global attention to scale the memory usage of self-attention modules linearly. ## experimental setup we demonstrate the effectiveness of our attention guidance loss through several empirical studies. specifically, we 1) report convergence results on masked language modeling, 2) evaluate trained language models on downstream tasks, and 3) analyze the learned attention representations using probes. for 1) and 2) above, we perform experiments on both high-resource and low-resource settings. ## conclusion in this study, we introduce the simple yet effective attention guidance (ag) loss, which speeds up convergence and improves performance on various domains and model sizes. adding this loss also makes transformers robust to hyperparameters like learning rate, warmup steps, and dropout. our experiments also show its usefulness in multiple pre-training objectives. the gains are particularly strong on larger models, enabling their usage in low-compute scenarios and low-resource domains. our analysis of the relation of ag loss and mlm loss shows the usefulness of our method, and we hope that this paper can serve as a starting point for future works aiming to exploit and question self-attention in transformers.