SO r T -ing VQA Models : Contrastive Gradient Learning for Improved Consistency (2021)

Abstract: Recent research in Visual Question Answering (VQA) has revealed state-of-the-art models to be inconsistent in their understanding of the world - they answer seemingly difficult questions requiring reasoning correctly but get simpler associated sub-questions wrong. These sub-questions pertain to lower level visual concepts in the image that models ideally should understand to be able to answer the reasoning question correctly. To address this, we first present a gradient-based interpretability approach to determine the questions most strongly correlated with the reasoning question on an image, and use this to evaluate VQA models on their ability to identify the relevant sub-questions needed to answer a reasoning question. Next, we propose a contrastive gradient learning based approach called Sub-question Oriented Tuning (SOrT) which encourages models to rank relevant sub-questions higher than irrelevant questions for an <image, reasoning-question> pair. We show that SOrT improves model consistency by up to 6.5% points over existing approaches, while also improving visual grounding and robustness to rephrasings of questions.

## introduction current visual question answering (vqa) models struggle with consistency. they often correctly answer complex reasoning questions, i.e, those requiring common sense knowledge and logic on top of perceptual capabilities, but fail on associated low-level perception questions, i.e., those directly related to the visual content in the image. for e.g., in fig 1 , models answer the reasoning question "was this taken in the daytime?" correctly, but fail on the associated perception question "is the sky bright?" indicating that the models likely answered the reasoning question correctly for the wrong reason(s). in this work, we explore the usefulness of leveraging information about sub-questions, i.e., low-level perception questions relevant to a reasoning question, and irrelevant questions, i.e., any other questions about the image unrelated to the reasoning question, to improve consistency in vqa. @xcite have studied this problem and introduced the vqa-introspect dataset that draws a distinction between higher-level reasoning questions and lower-level perception sub-questions. we augment this dataset with additional perception questions from the vqav2 dataset such that @xmath0, reasoning @xmath1, which we refer to as sub-questions (e.g.,"is the sky bright?" in fig 1 ) and irrelevant perception questions, which we refer to as irrelevant questions (e.g., "is the train moving?" in fig 1 ) throughout this paper. we use gradient-based class activation mapping (grad-cam) vectors @xcite -a faithful function of the model's parameters, question, answer and image -to propose an interpretability technique that determines the questions most strongly correlated with a reasoning question for a model. this is measured by ranking questions based on the cosine similarity of their grad-cam vectors with that of the reasoning question. we find that top-performing vqa models often rank irrelevant questions higher than relevant questions. motivated by this, we introduce a new approach based on contrastive gradient learning to fine-tune a vqa model by enforcing relevant sub-questions to be ranked higher than irrelevant questions while answering a reasoning question. this is achieved by forcing the cosine similarity of the reasoning question's grad-cam vector with that of a sub-question to be higher than with that of an irrelevant question. we find that our approach improves the model's consistency, defined as the frequency with which the model correctly answers a sub-question given that it correctly answers the reasoning question. additionally, we assess the effects of our approach on visual grounding by comparing grad-cam heatmaps with human attention maps collected in the vqa-hat dataset @xcite . we find that our approach of enforcing this language-based alignment through better ranking of sub-questions also improves visual grounding. we also demonstrate that training vqa models by aligning grad-cam vectors helps in improving robustness to rephrasings of questions, as evaluated on the vqa-rephrasings dataset @xcite . ## related work visual question answering. the vqa task @xcite requires answering a freeform natural language question about visual content in an image. previous work has shown that models often do well on the task by exploiting language and dataset biases @xcite @xcite @xcite . in order to evaluate the consistency of models, selvaraju et al. ## discussion in this work, we seek to improve consistency in vqa. we first develop language-based interpretability metrics to measure the relevance of a lower-level perception question while answering a higher-level reasoning question. evaluating stateof-the-art vqa models on these metrics reveals that models often rank irrelevant questions higher than relevant ones. we present sort (sub-question oriented tuning), a contrastive gradient learning based approach for teaching vqa models to distinguish between relevant and irrelevant perceptual concepts while answering a reasoning question. sort aligns grad-cam vectors of reasoning questions with those of sub-questions, while distancing them from those of irrelevant questions. we demonstrate sort's effectiveness on datasets that test for semantic as well as syntactic consistency without major changes to accuracy, while also improving visual grounding. ## acknowledgements the georgia tech effort was supported in part by nsf, afrl, darpa, onr yips, aro pecase, amazon. the views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of the u.s. government, or any sponsor. ## ethical considerations the key ethical considerations for this work relate to fairness. although not ubiquitous in application today, the progress of research in vqa necessitates work in the direction of transparency so as to build trust among users before these systems are widely deployed in the real world. prior work in this domain has revealed vqa models to exploit visual and language based priors in the datasets they are trained on @xcite @xcite @xcite . such models tend to compound the biases prevalent in these datasets, and could have detrimental effects on fairness. our work could better explain these biases by identifying the most relevant perceptual concepts used by the model while answering reasoning questions. in addition, by improving consistency and visual grounding in vqa systems, our work contributes to mitigating some of these biases.