A negative case analysis of visual grounding methods for VQA (2020)

Abstract: Existing Visual Question Answering (VQA) methods tend to exploit dataset biases and spurious statistical correlations, instead of producing right answers for the right reasons. To address this issue, recent bias mitigation methods for VQA propose to incorporate visual cues (e.g., human attention maps) to better ground the VQA models, showcasing impressive gains. However, we show that the performance improvements are not a result of improved visual grounding, but a regularization effect which prevents over-fitting to linguistic priors. For instance, we find that it is not actually necessary to provide proper, human-based cues; random, insensible cues also result in similar improvements. Based on this observation, we propose a simpler regularization scheme that does not require any external annotations and yet achieves near state-of-the-art performance on VQA-CPv2.

## introduction visual question answering (vqa) @xcite , the task of answering questions about visual content, was proposed to facilitate the development of models with human-like visual and linguistic understanding. however, existing vqa models often exploit superficial statistical biases to produce responses, instead of producing the right answers for the right reasons @xcite . the vqa-cp dataset @xcite showcases this phenomenon by incorporating different question type/answer distributions in the train and test sets. since the linguistic priors in the train and test sets differ, models that exploit these priors fail on the test set. to tackle this issue, recent works have endeavored to enforce proper visual grounding, where the goal is to make models produce answers by looking at relevant visual regions @xcite ## existing vqa methods given a question q and an image i, e.g., represented by bottom-up region proposals: v (anderson et al., 2018), a vqa model is tasked with predicting the answer a: ## embarrassingly simple regularizer the usage of visual cues and sensitivities in existing methods is superfluous because the results indicate that performance improves through degradation of training accuracy. we hypothesize that simple regularization that does not rely on cues or sensitivities can also achieve large performance gains for vqa-cp. to test this hypothesis, we devise a simple loss function which continuously degrades the training accuracy by training the network to always predict a score of zero for all possible answers i.e. produce a zero vector (0). the overall loss function can be written as: note that this regularizer continually penalizes the model during the course of the training, whether its predictions are correct or incorrect. as shown in it is also interesting to note that the drop in training accuracy is lower with this regularization scheme as compared to the state-of-the-art methods. of course, if any model was actually visually grounded, then we would expect it to improve performances on both train and test sets. we do not observe such behavior in any of the methods, indicating that they are not producing right answers for the right reasons. ## discussion on proper grounding while our results indicate that current visual grounding based bias mitigation approaches do not suffice, we believe this is still a good research direction. however, future methods must seek to verify that performance gains are not stemming from spurious sources by using an experimental setup similar to that presented in this paper. we recommend that both train and test accuracy be reported, because a model truly capable of visual grounding would not cause drastic drops in training accuracy to do well on the test sets. finally, we advocate for creating a dataset with ground truth grounding available for 100% of the instances using synthetically generated datasets @xcite @xcite @xcite , enabling the community to evaluate if their methods are able to focus on relevant information. another alternative is to use tasks that explicitly test grounding, e.g., in visual query detection an agent must output boxes around any regions of a scene that match the natural language query @xcite . ## conclusion here, we showed that existing visual grounding based bias mitigation methods for vqa are not working as intended. we found that the accuracy improvements stem from a regularization effect rather than proper visual grounding. we proposed a simple regularization scheme which, despite not requiring additional annotations, rivals state-of-theart accuracy. future visual grounding methods should be tested with a more comprehensive experimental setup and datasets for proper evaluation. ramprasaath r selvaraju, stefan lee, yilin shen, hongxia jin, shalini ghosh, larry heck, dhruv batra, and devi parikh. 2019. taking a hint: leveraging explanations to make vision and language models more grounded. in proceedings of the ieee international conference on computer vision (iccv), pages 2591-2600. robik shrestha, kushal kafle, and christopher kanan. 2019. answer them all! toward universal visual question answering models. in the ieee conference on computer vision and pattern recognition (cvpr). bernard l welch. 1938. the significance of the difference between two means when the population variances are unequal. biometrika, 29(3/4):350-362. jialin wu and raymond mooney. 2019. self-critical reasoning for robust visual question answering. in advances in neural information processing systems (neurips), pages 8601-8611.