Adaptive Transformers for Learning Multimodal Representations (2020)

Abstract: The usage of transformers has grown from learning about language semantics to forming meaningful visiolinguistic representations. These architectures are often over-parametrized, requiring large amounts of computation. In this work, we extend adaptive approaches to learn more about model interpretability and computational efficiency. Specifically, we study attention spans, sparse, and structured dropout methods to help understand how their attention mechanism extends for vision and language tasks. We further show that these approaches can help us learn more about how the network perceives the complexity of input sequences, sparsity preferences for different modalities, and other related phenomena.

## introduction learning richer representations from visual and text data is a central task to solve multi-modal learning. attention-based methods have proven to be very useful in learning long term dependencies and forming richer representations of the input sequences. numerous approaches @xcite @xcite have been proposed for learning visiolinguistic representations with transformers. although these approaches have provided us with significant improvement on various benchmarks (language and visiolinguistic), the architectures used are overparameterized require extensive training lasting for several weeks using multiple objectives to form a generalized representation of the task to be addressed, which is then followed by fine-tuning on a downstream task. this workflow has become a concerning problem. it results in deep learning methodologies being inaccessible and increased carbon footprints @xcite . in this work, we specifically explore adaptive methods. we refer to adaptive mechanisms as those methods that change their behavior during training/run time and adapt stochastically to the environment based on data heuristics (parameters) learned by encountering samples from the same data distribution optimized by an objective function. alternative approaches such as pruning, distillation @xcite and quantization are rigid to some extent and induce some form of permanent modifications to the model. adaptive methods enforce the network to learn parameters such that their behavior changes as per the complexity of the input sequence as perceived by the neural network. the code to reproduce the results in this work is publicly available at this link foot_0 . current self-attention approaches assume that the attention span of a head is invariant to the complexity of an input sequence. attention heads can learn their optimal context size @xcite , which results in a reduction of flops. when an optimal attention span is learned, the amount of attention given to a particular input sequence by an attention head is determined by its context size. we show that the context size varies with the emergent complexity of the sequence, and spans can help us understand how much sensitive a layer is to an input sequence. training models with a quarter of a million parameters are not feasible and practical for most users. one effective way to facilitate neural network scaling is by making the weights of the network sparse. this configuration allows us to perform faster training of deeper networks with relatively less compute. to make attention distributions sparse, we use α entmax @xcite to obtain probability distribution of weights. normalized exponential functions like softmax cannot assign a zero attention weight. this property en-forces the context vector to stay dense, resulting in non-relevant sequences to be considered even though the network has discarded them by putting a deficient weight. adaptive sparsity can make an attention head to learn richer distributions by oscillating the behavior of distribution to stay between softmax and sparsemax. we show that this behavior can help us understand preferences for the density of attention weight distribution and how it varies amongst each head about different modality. we also study a form of regularization method called layerdrop @xcite to understand its regularization impact for multi-modal features. if the network can learn to drop identical layers (data driven pruning), then it can be regarded as an adaptive depth mechanism. we specifically use the every other pruning method where the user specifies the drop rate because it offers maximal gains as suggested compared to its counterpart pruning methods. this method has proven to be effective in reducing the number of parameters and pruning layers during inference. the contribution of this work is as follows: ## experimental setup visual question answering to solve the vqa task, given an image and a question related to it, the network is supposed to predict the right answer from the given set of answer choices. we performed all the experimentation on the vqa 2.0 dataset @xcite . the dataset consists of three sets with a train set containing 83k images and 444k questions, a validation set containing 41k images and 214k questions, and a test set containing 81k images and 448k questions. in this case, the network is asked to predict an answer from 3129 answer choices for a particular question. implementation we use the pre-trained weights provided by @xcite . we fine-tune lxmert to form visiolinguistic representations based on image and text sequences with adaptive approaches mentioned above. this operation is followed by a classifier that receives the concatenated pooled features of image and text to predict the answer. fine-tuning is performed on a single p100 gpu with 128 batch size. optimization is performed with lookahead @xcite with lamb @xcite as the inner optimizer. learning rate schedule is regulated by cyclical lr @xcite , with base and max learning rates set to 1e -5 and 1e -4. ## experimental findings and results adaptive span for understanding the complexity of the input sequence we demonstrate how learning spans can help in understanding the behavior of individual layers. figure 1 shows how span varies amongst different attention layers. studying spans can help us understand which layers are more sensitive to the input sequences encountered during the training process. in the case of single modality encoder, spans for self-attention layers for vision and language decrease monotonically, indicating that the learning behavior is somewhat similar, although slopes tell us that the rate of learning is dissimilar. similar behavior is seen in the cross-modality encoder for language. requiring a larger context size is indicative of the complexity of the sequences. when selfattention attends to both modalities, we observe that the intermediate layers responsible for forming complex representations increase their spans. this observation shows that a more significant span is necessary to attend both modalities jointly. selfattention also requires a high span when attending to visual features in the cross-modality encoder. this observation shows that visual sequences are perceived as a more complex input to process than a language input in the cross-modality encoder. determining sparsity preferences for vision and language modality with α the value of α determines if the head is favoring sparse or dense attention weight distribution. for dealing with language modality, self-attention favors mostly sparse mapping of attention weights in intermediate layers. similar behavior is observed inside cross-modality encoder as well. this observation shows that language modality benefits from sparse weights being assigned as attention distribution. the value of α is restricted below 1.5 for processing visual inputs. when vision modality is involved, heads that preferred sparse mapping initially are converging towards denser mapping, indicating that this representation of attention weights is preferred. we also observe that when both modalities are involved, the network prefers, even more, denser weight distribution. this observation shows that vision modality is given more preference (partly due to perceived complexity) over language inputs to process the sequence. figure 3 shows variation of α values as training progresses. regularization effect of layerdrop we consider two configurations of the model. the first one has 10 language, 6 vision, and 6 cross-modality layers with drop rate (p) set to 1 layer. in this case, the number of parameters is more, but the flops is equivalent to the standard 9-5-5 baseline configuration. the later one has the 9-5-5 configuration with p set to 1. this rate causes a flop reduction of 17.54%. it is observed that layerdrop requires ∼3.5x more compute runtime for convergence during training. a possible explanation can be that additional training aids in forming a consolidated understanding of multi-modal representations. even after ensuring the convergence of the model, a strong regularization effect (with a minimum value of p) prevents the network from achieving performance that is close enough with the mentioned adaptive methods with an equivalent number of parameters being used training. figure 2 and ## ablation analysis we normalize attention scores with entmax instead of softmax before applying the masking function to use both adaptive attention span and sparse attention weights mapping. it is evident from ## conclusion while attention-based approaches are becoming universal, computationally efficient ways must be favored for broader adoption of provided pretrained models on low resource hardware. adaptive methods can significantly reduce the cost incurred to train such models and carbon footprints. in this work, we extend adaptive approaches to visiolinguistic tasks to understand more about attention and adaptive mechanisms. while the empirical results are encouraging, important future work includes explorations of higher efficient adaptive and sparse mechanisms that can significantly cause flops and parameter reduction with minimal loss in performance.