How Vision Affects Language: Comparing Masked Self-Attention in Uni-Modal and Multi-Modal Transformer (2021)

Abstract: The problem of interpretation of knowledge learned by multi-head self-attention in transformers has been one of the central questions in NLP. However, a lot of work mainly focused on models trained for uni-modal tasks, e.g. machine translation. In this paper, we examine masked self-attention in a multi-modal transformer trained for the task of image captioning. In particular, we test whether the multi-modality of the task objective affects the learned attention patterns. Our visualisations of masked self-attention demonstrate that (i) it can learn general linguistic knowledge of the textual input, and (ii) its attention patterns incorporate artefacts from visual modality even though it has never accessed it directly. We compare our transformer’s attention patterns with masked attention in distilgpt-2 tested for uni-modal text generation of image captions. Based on the maps of extracted attention weights, we argue that masked self-attention in image captioning transformer seems to be enhanced with semantic knowledge from images, exemplifying joint language-and-vision information in its attention patterns.

## introduction recently, we have seen a surge of interest in explainability research for large-scale neural networks, e.g. transformers @xcite . a lot of the existing literature focuses on the analysis of attention @xcite in terms of linguistic knowledge it encodes @xcite . @xcite show that attention heads' patterns in bert @xcite resemble syntactic dependencies present in the text. they also use a probing classifier to identify how knowledge of syntax is distributed between attention heads. @xcite ; @xcite have shown that visualising the structure of attention in transformer models can help us see which parts of the model capture specific syntactic knowledge. @xcite demonstrate that not all attention heads are equally suitable for learning syntactic information. thus, pruning such heads can be an option to reduce the model's complexity. while attention is not always an explanation @xcite , some work @xcite has shown that extra fine-tuning on a syntax-related task can guide the model's attention to truly resemble syntactic information about the text. other approaches to the model's interpretability include, for example, a work by @xcite , which inspects how knowledge is transferred on the neuron level rather than attention level. while most of the existing research has placed the problem of model's explainability in the context of uni-modal text-based tasks, e.g. machine translation, the field of language-and-vision is somewhat lacking similar analysis for models trained to solve multi-modal tasks. this becomes especially important with the increasing interest in adopting transformers for learning better cross-modal representations @xcite . in addition, using large-scale models to improve grounding between language and vision representations @xcite requires vigilance regarding how information is learned in different parts of such densely structured models. multi-modal transformers are required to not only learn to perform symbol grounding, e.g. mapping natural language symbols into visual representations as defined by @xcite and a language model, but also learn to fuse information from two modalities, the nature of which has been an open question in the field @xcite @xcite . the effect that such multi-modal representations have on the attention in large-scale models has not been addressed a lot in the languageand-vision literature. more specifically, we need a better understanding of how self-attention in transformer processes the multi-modal information. in this paper, we analyse the masked selfattention part of the image captioning transformer, which performs a standard language masking task based on the textual input, and compare its attention patterns with masked attention in distilgpt-2, a textonly transformer. our goal is to identify what kind of knowledge is captured in representations learned by this part of the model and whether it is affected in any way by the visual modality, which is not directly accessible for this particular self-attention. we aim to answer the following questions: in addressing these questions, we believe that we show novel insights into how the information is transferred between inner self-attentions of complex architectures such as a transformer and how representations from specific components of such models are affected by the training objective and multi-modality. ## model fig. 1 shows the architecture of the image captioning transformer that we use for our experiments, first introduced by @xcite and built on top of the basic image captioning transformer @xcite . this architecture resembles many parts of the classic transformer @xcite , which was initially introduced for machine translation, consisting of three multi-head self-attention mechanisms. the standard transformer's encoder learns representations of the input text by passing it through two sublayers: multi-head self-attention and feed-forward network. each sub-layer has a residual connection around itself, followed by layer-normalisation operation. the decoder contains masked self-attention, which is used to learn linguistic knowledge of the the image is first passed through a pre-trained object detector to extract visual and geometric features. the left side self-attention (image encoder) consists of attention heads, where each of them utilises both visual and geometry information. on the right side, the masked self-attention (text encoder) is given the embeddings of the caption words and their positional information. the words are fed to the text encoder in an auto-regressive manner, e.g. one word at a time plus all the preceding words. the cross self-attention uses keys k and values v from the visual encoder, while queries q are coming from the textual encoder and finally predicts the output probabilities of the next word. once we reformulate the model's task from machine translation to image captioning (fig. 1 ), we naturally change the encoder's inputs. instead of the source sentence, the encoder uses representations of objects from the image as its input. on the decoder's side, the ground-truth captions that the model learns to generate are used as inputs during training. to prepare inputs to our encoder, we first extract visual features of the detected objects @xmath0, ..., x n }, where x @xmath1. we use a bottom-up feature extractor @xcite , which is based on faster-rcnn @xcite and pretrained on visual genome @xcite with the resnet-101 as its backbone @xcite . for each detected object we also extract geometry features @xmath2, y, w, h) (centre coordinates, width, height). in the next step, queries @xmath3: then, ω v and geometric features g are combined, taking into account the displacement between the objects and producing a fused representation ω. foot_0 finally, each attention head h from each encoder layer l outputs a combination of values v and geometry-aware visual features ω: masked self-attention in the decoder the idea of self-attention is that each token from the input text learns to attend to the other tokens from the same sequence. however, this is not feasible for the caption generation task since attending to the future tokens is unfair and it cannot be used when generating text. therefore, the self-attention in the decoder is using masking of future tokens to keep the autoregressive nature of the model. in particular, the token w t and the future tokens w t+1 , ..., w w are replaced with [m ask]. then, w t is predicted using the previous context in the standard left-to-right fashion: w \t := (w 1 , ..., w t-1 ). we have specifically focused on the analysis of the attention weights in the decoder's masked selfattention of the image captioning transformer. we extract the attention weights for each head h in each layer l of this self-attention and use them for our visualisations and analysis. these weights are calculated similarly to the attended visual features (eq. 1). our masked self-attention has six layers, consisting of eight heads in each of them. for the model checkpoint, we use the best model released by the authors of the architecture foot_1 . this checkpoint has been chosen on the basis of automatic evaluation scores: the model uses bottom-up representation of images, geometry features and self-critical training @xcite . the captions are generated using beam search with beam width @xmath4. ## learning syntactic knowledge in our first experiment we investigate whether the attention weights of the masked self-attention are able to capture any general syntactic knowledge about the input text. it has been shown that the multi-head attention patterns in the transformer trained for the task of machine translation resembles syntactic properties of language at the level of part-of-speech tags and syntactic dependencies @xcite . since the self-attention that we are focused on is trained in a very similar task (masked language modelling), we first explore if particular layers and heads attend to specific part-of-speech tags the most. then, we continue with the analysis of how information about syntactic dependencies is reflected in the learned attention patterns. attention on part-of-speech we follow @xcite who compute the proportion of attention from each head that this head pays to tokens of a particular part-of-speech tag and accumulate the results over our test set: the results for the five most frequently occurring part-of-speech tags (more than 1000 individual instances) are shown in fig. 2 . words of such partof-speech tags, which can be grounded in visual signals (nouns for objects, adjectives for attributes), receive attention from a large number of attention heads. on the other hand, only specific heads focus on words describing relations (verbs, adpositions). specifically, seventeen heads (out of forty-eight) put more than 40% of their attention to the nouns, while only three heads give more than 30% of their attention to the verbs. we also find supporting evidence for the previous studies @xcite , showing that deeper layers focus on more complex properties, e.g. relational part-of-speech tags (verbs), which require knowledge of objects learned from earlier layers (nouns). for example, the top 3 attention heads that attend to basic partsof-speech such as determiners are all located in the model's first three layers. for adjectives, the top 3 heads are similarly located in the first three layers of the model, with the maximum value of the attention head being 0.25. however, attention on adjectives is more spread across many heads in different layers, with the attention value being 0.14 for more than half of the heads, which is also a mean value for attention on this part-of-speech tag. a less clustered pattern is observed for nouns: its top 3 heads are located in layers 1, 3, and 4, with thirty-three heads paying more than 30% of their attention to nouns. we argue that the reason why the attention on nouns is scattered over many heads, with most of them paying nearly one-third of their attention to the nouns, is because nouns are continuously required for caption generation: the model needs to take them into account when generating either a relation or an attribute. somewhat differently, verbs are attended mostly in the model's deeper layers: the top 3 most attentive heads are located in layers 3, 4, and 5 with values higher than 0.3. the vast majority of the heads (forty-three) have smaller attention values (less than 0.2), indicating that the model needs verbs only for specific situations, for example when a relationship needs to be generated. overall, our visualisations demonstrate that masked self-attention weights resemble task-specific syntactic information about part-of-speech tags. for example, nouns are similarly attended across all heads since they are required for the captioning task the most (to describe, refer to, use in phrases, etc.). in contrast, more function-dependent parts of speech (verbs, adpositions) are attended to by fewer heads in the deeper layers of the model. attention on syntactic dependencies fig. 3 shows the proportion of attention from the heads in masked self-attention for the most frequently occurring syntactic dependency relations. the proportions are calculated similarly to eq. 3. in particular, we used the attention weights from root to the nonroot part of the dependency phrase or vice versa, extracting dependencies in advance. this choice was affected by the auto-regressive nature of the generation task: for each word, we could only inspect attention focus on previous words. the attention on different dependencies seems to be distributed similarly to the attention on part-of-speech tags. more specifically, attention heads from the surface layers (1:5 and 2:1 foot_2 ) seem to be focused on the determiner in the det relation. comparing heatmaps of attention distribution on part-of-speech and syntactic dependency may give us intuition figure 3 : attention distribution on different constituents of the specific syntactic dependencies. for det, compound, nummod we visualise which heads look the most on the non-root element of the dependency (e.g. "man" → "a" in "a man"). for pobj and prep we show attention in a different direction (e.g. "table" → "on" in "on table", "with" → "bathroom" in "bathroom with"). about the specific heads' role. for example, the heads 3:4 or 3:5 are not intensely active for the det relation, although they are among the most active heads when attending to the determiners. this indicates that these heads 1:5 and 2:1 may be more responsible for focusing on determiners when the phrase in the det relation is generated. interestingly, many heads strongly attend to the numeral in the nummod dependency compared to all other relations. this could be related to the importance of learning about the number of objects in the scene, while other, simpler noun-based dependencies (det, compound) do not have to be attended so strongly. only a few heads specialise in dependencies that capture more complex properties (e.g. relations between different objects), with heads 3:3 and 3:6 being the most attending heads for pobj. the root of the prep phrase is often attended in the first layer, with only a few more heads in the later layers being activated. could this pattern be mapped with the fact that roots in these phrases are often nouns and verbs? fig. 4 shows that heads 1:3 and 1:7 are the most active heads when a noun is a root in the phrase of prep dependency. same heads in the first layer are also active the most when looking at the nouns, according to fig. 2b . this indicates that the model acquires basic knowledge of language syntax (dependencies, part-of-speech information) in its first layers. similarly, as fig. 4b demonstrates, the head 3:6 is the single most active head for the prep dependency. at the same time, according to fig. 2d , this particular head is one of the few most active heads when the attention focus is on verbs. this might be interpreted as if this head is better at learning information about syntactic dependencies than other activated heads. we argue that it is helpful to look at the correspondence between attention on parts-of-speech and syntactic dependency since it is informative when determining specific heads' roles and how important they are for different language tasks, e.g., part-of-speech tagging and syntactic dependency identification. ## multi-modality and masked self-attention in this section, we look at how a multi-modal task of image captioning affects attention on the previous words when a masked self-attention model predicts the next word. we also compare our model's attention patterns with patterns from an auto-regressive model, distilgpt-2 @xcite , which has been pre-trained on openweb-textcorpus. this model has 6 layers with 12 heads in each layer, which makes it more comparable to our captioning transformer than the standard gpt-2 model with 12 heads in each of the 12 layers. ## attention alignment it may be the case that the observed differences in attention patterns discussed in the previous section are simply due to different frequencies of words (in particular nouns) in the dataset on which the models are trained. for example, the multi-modal decoder also attends on the closest syntactic relations in the same way as a uni-modal decoder, but these happen to be nouns simply because there are more nouns in image captions. to test this hypothesis we calculated the pearson correlation coefficient between the frequency of the nouns in our captions and attention distribution on the context words attended by heads when the next word is produced. the test has not shown a statistically significant correlation between the frequency of the nouns versus attention distribution on the context words in multi-modal decoder's self-attention (@xmath5.49, @xmath6.056). however, we observed a moderate positive correlation between the frequency of the nouns versus attention distribution in the uni-modal decoder's attention (@xmath7.60, @xmath8.014). these differences in correlations show that the uni-modal architecture is more biased to frequencies, whereas in a multi-modal setting, the effect of noun frequency is diminished. this provides support to our hypothesis, namely that this bias towards nouns is coming from somewhere else, e.g. the multi-modal representations that the language model is grounded in. since the model's parameters are jointly updated with an end-to-end training through backpropagation, representations learned by different self-attention mechanisms are expected to be aligned with each other. we present a small preliminary analysis of whether the attention weights in the cross-modal self-attention (cross self-attention from fig. 1 ) are responsible for information fusion between image encoder and text decoder. our hypothesis is as follows: if cross-modal self-attention pays a significant portion of attention to the objects, which are generated as nouns in the caption as content words, we can conclude that due to the learning objective and nature of the information flow within the model's components, decoder's selfattention aligns with a higher-level cross-modal self-attention. in this case, we also expect that for every non-content word (e.g., determiner, preposition), the cross-attention keeps its attention on the most recent content word similar to what we observe for decoder's self-attention in figs. 5e-5h. we use two example images and examine the differences among the top 5 most-attended objects for every word generated in image captions. we use the predicted labels from the feature extractor @xcite to refer to the detected objects. fig. 8 shows changes in cross-modal attention on objects during generation of descriptions. from fig. 8a we can see that every time a new content word is generated ("plate", "sandwich", "salad"), the cross-modal attention tends to focus on objects with labels that are similar to the generated content words. for example, "lettuce" and "salad" are among the most attended objects when the transformer is preparing to generate the content word "salad". also, the same objects are continued to be attended when other non-content words are generated. this example provides initial evidence how text generation of nouns as exemplified by the decoder's attention is linked to multi-modal representations as exemplified by cross-modal attention on objects. the results suggest that in multimodal settings models learn representations that are fused and aligned with each other. since the selfattention in the uni-modal architecture only needs to generate the text one word at a time by taking into account only previously generated words, it learns a pattern over local syntactic dependencies. in our future work, we would like to provide a more detailed analysis of the cross-modal attention and the uni-modal visual attention and therefore further strengthen the arguments how multi-modality affects knowledge that different parts in the large scale transformer models learn. ## conclusion we have shown that attention patterns learned by a sentence decoder module of a multi-modal transformer are highly affected by the task that the model is optimised for. we focused on the masked self-attention in a sentence decoder in an image captioning transformer, demonstrating that its attention weights resemble linguistic knowledge, which is affected by the task of image captioning. this indicates that such language model acquired important aspects of grounded semantics. simultaneously, we show that that it is important to be cautious when applying large-scale pre-trained models on specific tasks to different semantic tasks as the original task does have an impact on the semantic representations learned. our future work will focus on further examination of self-attention in the other two components of the multi-modal models which will give us an even clearer picture on what representations are learned by them.