VQA - MHUG : A Gaze Dataset to Study Multimodal Neural Attention in Visual Question Answering (2021)

Abstract: We present VQA-MHUG – a novel 49-participant dataset of multimodal human gaze on both images and questions during visual question answering (VQA) collected using a high-speed eye tracker. We use our dataset to analyze the similarity between human and neural attentive strategies learned by five state-of-the-art VQA models: Modular Co-Attention Network (MCAN) with either grid or region features, Pythia, Bilinear Attention Network (BAN), and the Multimodal Factorized Bilinear Pooling Network (MFB). While prior work has focused on studying the image modality, our analyses show – for the first time – that for all models, higher correlation with human attention on text is a significant predictor of VQA performance. This finding points at a potential for improving VQA performance and, at the same time, calls for further research on neural text attention mechanisms and their integration into architectures for vision and language tasks, including but potentially also beyond VQA.

## introduction visual question answering (vqa) has gained popularity as a practically-useful and challenging task at the intersection of natural language processing (nlp) and computer vision (cv) @xcite . the key challenge in vqa is to develop computational models that are able to reason over questions and images in order to generate answers that are well-grounded in both modalities (p. @xcite @xcite . attention mechanisms originally introduced in nlp for monomodal language tasks have been successfully applied to multimodal tasks (like vqa) and established a new state of the art @xcite @xcite . these advances have, in turn, triggered research into understanding the reasons for these improve-ments. a body of work has studied similarities between neural and human attention @xcite @xcite . models seem to learn very different attention strategies and similarity to human attention might only improve performance for specific model types @xcite . however, although vqa is an inherently multimodal task, all of these analyses have only focused on image attention. the most likely reason for this is that existing datasets only offer monomodal attention on the image @xcite @xcite . in addition, due to the challenges involved in recording human gaze data at scale, prior works have instead used mouse data as a proxy to attention @xcite . however, mouse data was shown to overestimate some image areas @xcite or to miss relevant background information altogether @xcite . as of now, there is no publicly available dataset that offers human gaze data on both the images and questions. this severely impedes further progress in this emerging area of research. our work fills this gap by introducing vqa-mhug -the first dataset of multimodal human gaze on both images and questions in vqa. to collect our dataset, we conducted a 49-participant eye tracking study. we used a commercial, highspeed eye tracker to record gaze data on images and corresponding questions of the vqav2 validation set. vqa-mhug contains 11,970 gaze samples for 3,990 question-image pairs, tagged and balanced by reasoning type and difficulty. we ensured a large overlap in question-image pairs with nine other vqa datasets to maximize the usefulness of vqa-mhug for future multimodal studies on human and neural attention mechanisms. using our dataset, we conduct detailed analyses of the similarity between human and neural attentive strategies, the latter of which we obtained from five top-performing models in the vqa challenges 2017-2020: modulated co-attention network (mcan) with grid or region features, pythia, bilinear attention network (ban), and the multimodal factorized bilinear pooling network (mfb). these analyses show, for the first time, that correlation with human attention on text is a significant predictor of accuracy for all the studied state-of-the-art vqa models. this suggests a potential for significant performance improvements in vqa by guiding models to "read the questions" more similarly to humans. in summary, our work contributes: 1. vqa-mhug, a novel 49-participant dataset of multimodal human gaze on both images and questions during visual question answering collected using a high-speed eye tracker. 2. detailed analysis of the similarity between human and neural attentive strategies indicating that human-like attention to text could yield significant performance improvements. our work is related to previous work on 1) neural machine attention, 2) attention in vqa, and 3) comparison of neural and human attention. ## the vqa-mhug dataset we present visual question answering with multi-modal human gaze (vqa-mhug) 1 . to the best of our knowledge, this is the first resource containing multimodal human gaze data over a textual question and the corresponding image. our corpus encompasses task-specific gaze on a subset of the benchmark dataset vqav2 val foot_1 @xcite . we specifically focused on question-image pairs that machines struggle with, but humans answer easily (determined by high inter-agreement and confidence in the vqav2 annotations). we then balanced the selection by evenly picking questions based on a machine difficulty score and from different reasoning types. thus, vqa-mhug covers a wide range of challenging reasoning capabilities and overlaps with many vqav2-related datasets (see reasoning types. vqav2 groups questionimage pairs based on question words: what, who, how, when and where. instead, we binned our pairs into the reasoning capabilities required to answer them. we incorporated the categories proposed by @xcite for their task directed image understanding challenge (tdiuc) and extended them with an additional category, reading, for questions that are answered by reading text on the images. this resulted in 12 reasoning types that align better with commonly-diagnosed error cases foot_2 . we binned vqav2 val pairs accordingly by training a lstm-based classifier on 1.6 m tdiuc and 145 k vqav2 train+val samples which we labelled using regular expressions. the classifier predicted the reasoning type for a given question-answer pair. the final model achieved 99.67% accuracy on a 20% held-out test set. machine difficulty score. to assess the difficulty for a machine to answer a question-image pair, we ran two popular vqa models -mfb @xcite for multimodal fusion and mcan @xcite for transformer attention -inspired by @xcite . a difficult question results in low answer accuracy, particularly after rephrasing or asking further control questions. to test this, we evaluated on four datasets and averaged their corresponding normalized metrics: (1) vqav2 accuracy, (2) vqa-cp accuracy on reduced bias @xcite , (3) vqa-introspect's consistency with respect to visual perception @xcite and (4) vqa-rephrasings' robustness against linguistic variations @xcite ## comparison of human and machine attention the collected data enabled us to analyze whether models achieve a higher accuracy on vqav2 val the more their attentive behavior over the text and image correlates with human ground-truth attention. hence, we investigated the attention weights over text and image features of different sota vqa models. ## conclusion and future work in this work we have presented vqa-mhug -a new, fully annotated 49-participant dataset for visual question answering that includes nearly 4,000 question-answer pairs. our dataset is unique in that it is the first to provide real human gaze data on both images and corresponding questions and, as such, allows researchers to jointly study human and machine attention. revealed through a detailed comparison of multiple leading vqa models, we showed that higher correlation between neural and human text attention is a significant predictor of high vqa performance. this novel finding highlights the potential to improve vqa performance with human-like attention biases and simultaneously calls for further investigation of neural text attention mechanisms, as we find these are an indicator for success on language and vision tasks, including vqa.