Understanding Attention for Vision-and-Language Tasks (2022)

Abstract: Attention mechanism has been used as an important component across Vision-and-Language(VL) tasks in order to bridge the semantic gap between visual and textual features. While attention has been widely used in VL tasks, it has not been examined the capability of different attention alignment calculation in bridging the semantic gap between visual and textual clues. In this research, we conduct a comprehensive analysis on understanding the role of attention alignment by looking into the attention score calculation methods and check how it actually represents the visual region’s and textual token’s significance for the global assessment. We also analyse the conditions which attention score calculation mechanism would be more (or less) interpretable, and which may impact the model performance on three different VL tasks, including visual question answering, text-to-image generation, text-and-image matching (both sentence and image retrieval). Our analysis is the first of its kind and provides useful insights of the importance of each attention alignment score calculation when applied at the training phase of VL tasks, commonly ignored in attention-based cross modal models, and/or pretrained models. Our code is available at: https://github.com/adlnlp/Attention_VL

## introduction the relative maturity and flexibility of deep learning allow us to build upon the success of computer vision and natural language processing to face many complex and multimodal vision-and-language (vl) tasks, such as visual question answering (vqa), text-and-image matching (t&i match), or text-to-image generation (t2i gen). for these vl tasks, it is crucial to effectively align the multimodal information in both visual and linguistic domains. for example, to pick the right answer in vqa, the model should empower information from the input image, together with aligning the linguistic meanings with visual clues. attention mechanism @xcite has been used as an important component across a wide range of vl models; from the early-stage attention-based fusion vl models @xcite @xcite to the recent vl multimodal transformer-based pretrained models @xcite @xcite . those attention-based vl models mainly focus on 1) exploring new features to represent visual and linguistic information as an input of attention layer, 2) deciding the position or the number of attentions in the model, or 3) investigating the interpretability of attention distribution on vl tasks by emphasising the specific image regions or textual tokens @xcite . while such approaches and investigations have resulted in interesting findings in different aspects of vl tasks, the attention alignment calculation between vision and language modalities has been less explored. however, the alignment calculation is directly linked to the main purpose of using attention mechanisms in vl tasks, which is to effectively bridge and align two different visual and linguistic information. in other words, the essence of the attention mechanism in vl tasks is the alignment score calculation, as it quantifies the amount of "attention" that the visual features would place on each of the language representations (or linguistic features would empower on the specific visual regions) when bridging the semantic gap between visual and language features. most existing vl models directly apply the two attention alignment functions, a general and a dot-product @xcite , which are commonly used in several nlp tasks. since @xcite proposed a scaled dot-product for the transformer with full attention, almost every vl paper has directly applied those three attention alignment score functions. instead, little work has been done towards understanding the role of attention alignment calculation methods applied to bridge visual and linguistic features, and exploring the impact on different vl model performance. in order to address this limitation, the overarching goal of this research is to perform an extensive and systematic assessment of the effect of a range of attention alignment mechanisms pertaining to vl tasks, including three major vl tasks: visual question answering (vqa), text-and-image matching (t&i match), and text-to-image generation (t2i gen). towards that end, we systematically analyse the impact of the position of query and key in attention alignment on vl tasks. we investigate the following three questions: i) which attention alignment score calculation yields the most benefit in vl tasks? ii) what if we linearly transform the query q instead of the key k (or vice versa) before the multiplication? for example, assume the textual feature t is a query q, and the image feature i is a key k. we analyse the impact of linear transforming q or k in alignment score calculation. iii) do the attention alignment calculation techniques with better performance provide better attention distribution interpretability? in brief, our main contributions are as follows: text-to-image generation attngan @xcite first proposed to use dot-product for measuring the alignment between visual subregions and word tokens to guide the image generation process. many of the later approaches directly adapted the dot-product attention from attngan @xcite @xcite @xcite @xcite . a few models apply the element-wise multiplication @xcite or cosine similarity @xcite for measuring the attention alignment. ## attention alignment mechanism there are various attention mechanisms applied in different multimodal vl downstream tasks. two commonly used approaches are the cross-attention and the self-attention. first, the cross-attention is performed between visual and textual inputs. more specifically, given a sequence of textual features @xmath0, t 2 , t 3 , . . . , t m } and image features @xmath1, i 2 , i 3 , . . . , i n }, it takes t as the query q and i as the key k (or vice versa) to compute attention and context vectors c as the attended representations of the input elements in the following way: scaled dot product attention the higher dimension of data representation would lead to the smaller gradient of softmax function. hence, the scaling factor was introduced by @xcite , and applied to the self-attention-based vl approaches as represented in equation 4 . general attention along with dot product attention, general attention @xcite received lots of interest as an alternative alignment calculation method that computes attention score using an extra learnable matrix to linearly transform k into the same embedding space as q. this can be considered as one of the neural network based methods mentioned in section 2. there are several variants of neural network based general attention calculation methods. first, biased general attention is introduced by @xcite using more bias towards more important keys regardless of the query context. secondly, activated general attention. @xcite applies an additional nonlinear activation term, which is able to amplify the emphasis on query elements that are highly relevant to the key. in this paper act is the relu activation since it is a widely used function in vl downstream tasks. ## vision-language models we use publicly available implementations of the most widely adopted vl baseline models foot_0 in order to train and evaluate different attention alignment score calculation for three different vl tasks: (i) attngan for text-to-image generation (t2i gen), (ii) scan for text-and-image matching (t&i match), (iii) mac and m4c for each visual question answering (vqa) and text-based visual question answering (tvqa). ## results we analyse the impact of attention alignment mechanisms in different vl tasks, and explore the interpretability based on attention distribution. ## conclusion we systematically examined the role of attention alignment score calculation in vision-and-language tasks, including vqa, t&i match, and t2i gen. we found that the scaled dot product can be the best attention alignment calculation for either cross or self-attention in overall vl tasks while the appropriate position of visual and textual information may vary from different vl tasks/datasets. based on the breakdown analysis, we found out that the type of image objects and their textual description would affect the performance of different attention calculation functions. it is hoped that our analysis provides a great insight into the selection of the most effective attention alignment for different vl tasks.