Uncovering the Full Potential of Visual Grounding Methods in VQA (2024)

Abstract: Visual Grounding (VG) methods in Visual Question Answering (VQA) attempt to improve VQA performance by strengthening a model’s reliance on question-relevant visual information. The presence of such relevant information in the visual input is typically assumed in training and testing. This assumption, however, is inherently flawed when dealing with imperfect image representations common in large-scale VQA, where the information carried by visual features frequently deviates from expected ground-truth contents. As a result, training and testing of VG-methods is performed with largely inaccurate data, which obstructs proper assessment of their potential benefits.In this study, we demonstrate that current evaluation schemes for VG-methods are problematic due to the flawed assumption of availability of relevant visual information. Our experiments show that these methods can be much more effective when evaluation conditions are corrected. Code is provided.

## introduction visual grounding (vg) in vqa has garnered interest not only as a key aspect to furthering understanding and rationalization of a vqa model's inference procedure, but also as a way to improve out-of-distribution (ood) performance by preventing certain dataset biases to form. various works have reported evidence of problematic tendencies in vqa models that point to a disregard of relevant image regions during answer inference and the manifestation of q/a distribution biases in the model @xcite @xcite . a lack of vg in vqa models has been shown to negatively impact ood performance @xcite and have been tied to a general unpredictability of answering behavior @xcite . to alleviate these issues, methods have been developed that seek to strengthen a model's reliance on question-relevant visual features. these vg-methods either modify the training procedure of existing models (e.g., hint @xcite , scr @xcite , visfis @xcite ), or are integrated directly into specialized model architectures such as mmn @xcite , pvr @xcite and vlr @xcite . on a technical level, the goal of vg-methods in vqa is to align a model's internal valuation of visual input feature importance (fi) with humanbased fi, which is given as guidance in training. these human-based fi scores can be inferred from a question's visual relevance annotations, which may be given as highlighted regions in the raw image (e.g., spatial heat maps as in vqa-hat @xcite ), or explicit pointers to ground-truth objects (as in gqa @xcite ). notably, relevance annotations are not given in input feature space directly, and therefore a mapping function is required to identify corresponding visual features and determine fi scores. the predominant approaches for such a mapping between image and feature space rely exclusively on spatial matching: visual input features receive their fi scores depending on spatial overlap between the region they represent and annotated, questionrelevant locations in the raw image @xcite ; @xcite ; @xcite ; @xcite ). high-scoring features can then be identified as relevant cue objects 2 . in this approach, the actual visual content carried by the cue objects is simply assumed to be appropriate without further semantic verification and therefore does not influence their score. in this work, we report evidence that such incomplete verification can result in grossly mismatched cues, thereby leading to inadequate guidance in vg-method training, as illustrated in fig. 1 , left. similarly, tests performed under such unchecked conditions fail to accurately evaluate the originally intended use case that vg-methods were designed for, as question-relevant content is often missing in the input and proper vg impossible. while work such as @xcite and @xcite investigate the underlying effects of vg-method application in detail, we are unaware of any study that also considers the impact of these problematic conditions in their analysis. in this study, we seek to develop a better understanding of the benefits of vg-methods in vqa when training and testing conditions properly support their intended use-case. we identify two flaws and their causes in current evaluation practices for vg-methods and outline an approach to fix them. finally, we investigate their impact on vg-method training and testing in a series of experiments. the used methodology establishes a framework for evaluating vg-methods more thoroughly. contributions. summarized as follows: in all cases, the image-based relevance annotations are subsequently used to identify relevant cue objects in a model's visual input feature space. the mapping from annotations to input features has traditionally been based entirely on the features' receptive field, i.e., the image location that the features represent @xcite @xcite . our work goes one step further and examines vg-methods that consider cue objects that additionally match the content of the relevant ground-truth object. to our knowledge, the effects of such semantic matching on vg-method efficacy in vqa has not been explicitly investigated before. ## flawed visual grounding we posit that evaluations of vg-boosting methods in vqa are following a flawed methodology, which consequently causes a flawed understanding of the benefits of vg and vg-boosting methods in vqa. specifically, we investigate the following issues: (f1) flawed testing: current evaluations of vgmethods hide their full potential by diluting tests with questions that are impossible to correctly ground due to missing relevant visual information. 3 (f2) flawed training: impact of vg-boosting methods is muted due to training with a large percentage of unsuitable training samples that are missing relevant visual information necessary for teaching consistently correct inference alignments. fig. 1 , left, illustrates this problem. we identify two underlying causes for these issues: (c1) noisy features: impacts f1 and f2. object misrecognitions and missing detections of relevant objects in the input image representation occur frequently in large-scale object detection tasks. we find that only 30% of the used training questions (and 27%/26% of id/ood tests) in our gqa experiments contain all necessary question-relevant information. (c2) fuzzy spatial matching of cue objects: impacts f2. spatial identification of relevant cue objects may declare irrelevant objects as relevant on account of their close vicinity to the reference location, even if the represented visual content is inadequate and therefore irrelevant (as illustrated in fig. 1 ). we identify a question-average of 2.6 cue objects in gqa 3 we illustrate the problem of flawed testing with two examples. consider the five solid colored squares in fig. 1 , left, as a model's visual input and the question as follows: 1) the question "is the truck's back door open?" requires vg on the truck, but the input does not contain a truck (missing object detection is signified by the red dashed square). 2) the question "what is the cow doing?" requires vg on the cow, but the input does not contain a cow (wrong object recognition is signified by the red colored square). in both cases, impact of proper vg on accuracy cannot be cleanly evaluated. training using semantic matching, which is inflated to 5.4 cue objects using spatial matching (counted based on a threshold of @xmath0.5; matching methods defined in sec. 4.2). this means that on average more than half of the objects that were declared question-relevant by spatial matching are in fact irrelevant. ## experimental setup we empirically show that addressing flaws f1 & f2 outlined in sec. 3 provides new insights into the efficacy of vg-methods in vqa. in this section, we describe our methodology for fixing these flaws for analytical purposes. ## impact on visual grounding quality following recommendations in @xcite , we investigate impact of vg-methods on a model's visual grounding quality with the dedicated metric fpvg @xcite . ## true vg analysis with vqa-hat in this section, we adopt the true vg methodology for the vqav1-based vqa-hat-cp datset. ## conclusion in this study, we have shown that current training and testing practices for vg-methods in vqa are flawed and therefore unable to reflect their full potential benefits for vqa models. we have proposed a methodology to optimize evaluation conditions to allow for a more thorough analysis of vg-method impact. our investigations have shown that when conditions are optimized, vg-methods can elicit considerably stronger performance improvements in vqa models in both vg and accuracy, boosting ood performance in particular.