Attending Self-Attention: A Case Study of Visually Grounded Supervision in Vision-and-Language Transformers (2021)

Abstract: The impressive performances of pre-trained visually grounded language models have motivated a growing body of research investigating what has been learned during the pre-training. As a lot of these models are based on Transformers, several studies on the attention mechanisms used by the models to learn to associate phrases with their visual grounding in the image have been conducted. In this work, we investigate how supervising attention directly to learn visual grounding can affect the behavior of such models. We compare three different methods on attention supervision and their impact on the performances of a state-of-the-art visually grounded language model on two popular vision-and-language tasks.

## introduction the introduction of transformers @xcite has been a major component of the success of pre-trained language models @xcite @xcite which achieved new records in many natural language processing tasks. the same mechanism has been adapted to create models @xcite @xcite @xcite ; lxm) that can now tackle vision-andlanguage tasks with impressive performances. a large body of research @xcite has been dedicated to understanding what attention heads learn during the pre-training of language models. @xcite have even shown how providing attention heads with guidance can improve performance on neural machine translation. on the other hand, the internal behaviors of vision-and-language models have attracted less interest from the research community. @xcite have shown some attention heads in vision-andlanguage models are able to map entities to image regions while others even detect syntactic relations between non-entity words and image regions. nevertheless, no initiative has been taken towards supervising directly the attention modules. in this paper, we study how different methods on attention supervision can affect vision-andlanguage models. we propose a fine-tuning method aimed at using the visual grounding of entities to provide guidance to attention heads. we compare three different methods by evaluating their performance on popular downstream tasks and visualize the different attention modules obtained. we observe that an indirect method which uses a module appended to the final output of the transformer obtains worse results than methods which focus on supervising every attention head directly. the codes are available at https://github.com/ jules-samaran/vl-bert . ## our method we use a state-of-the-art pre-trained vision-andlanguage model on which we propose multi-task fine-tuning methods focusing on attention supervision. after this proposed fine-tuning, we judge the success of our approach by further fine-tuning the model on downstream tasks of visual question answering and referring expressions, and evaluating it. we propose a fine-tuning approach after an initial pretraining step on a large unlabelled dataset because we believe the model would benefit from learning first from scratch freely about text and images without any supervision on its attention heads, and that our fine-tuning would allow it to then refine the representations it provided using visual grounding labels. ## related work vision and language pre-trained models on large image caption datasets have been proposed such as @xcite , vilbert @xcite , vl-bert @xcite , lxmert (lxm) and uniter @xcite . those vision and language pretraining models differ from the model architecture. we study visually grounded attention supervision in vl-bert. @xcite ; @xcite analyzed on language pre-trained models and showed that different attention heads share similar patterns and behaviors. for neural machine translation, @xcite proposed to use word alignment for cross-attention supervision during decoding in a recurrent neural network based architecture. we work specifically on vision-and-language transformers and use phrase visual grounding for attention supervision in order to help the model learn how to align phrases with their associated regions in the images. ## conclusion motivated by similar works in language models, we have presented three different methods that attempt to guide the model in its learning of entity grounding. we observed that the indirect method which is the most similar to the structure used for downstream tasks had a little or negative effect on the performance of the model. we also found that supervising attention heads through attention roll-out is the best performing method nevertheless all these methods fell short of the performances of the model before being fine-tuned on the vgp dataset. despite the performance, we have shown which attention supervision methods give better results and more interpretable attention patterns foot_1 (i.e., direct and semi-direct attention) than others that should not be used (i.e., indirect attention). therefore, we believe that our work can pave the way for further analyses of how this mechanism could be made to improve the performance of vision-andlanguage models. for future work, we plan to study how direct supervision methods could be applied on some selected heads instead of supervising uniformly all attention heads in every layer.