A Multi-modal Debiasing Model with Dynamical Constraint for Robust Visual Question Answering (2023)

Abstract: Recent studies have pointed out that many well-developed Visual Question Answering (VQA) systems suffer from bias problem. Despite the remarkable performance gained on In-Distribution (ID) datasets, the VQA model might merely capture the superficial correlation from question to answer rather than showing real reasoning abilities. Therefore, when switching to Out-of-Distribution (OOD) dataset, whose test distribution is unknown or even reversed with the training set, significant drops might be demonstrated. Although efforts have been devoted to easing the negative bias effect brought by language prior and analysing its inherent cause, they are still limited by the following two aspects. First, most current debiasing methods achieve promising OOD generalization ability with a major sacrifice of the ID performance. Second, existing researches are restricted by exploiting comprehensive biases, since weakening the language bias is mainly focused, while only a few works consider vision bias. In this paper, we investigate a straightforward way to mitigate bias problem for VQA task. Specifically, we reduce bias effect by subtracting bias score from standard VQA base score. Based on such a direct strategy, we design two bias learning branches to detect more bias information, which are combined with a dynamical constraint loss to alleviate the problem of over-correction and insufficient debiasing effect. We evaluate our method on the challenging VQA v2.0 and VQA-CP V2,0 datasets and the proposed method achievessignificant improvement.

## introduction visual question answering (vqa) @xcite is a challenging task spanning both computer vision and natural language processing. the goal of vqa is to infer the answer based on a given image and a textual question, which is generally cast as a classification problem. promising results on test set whose distribution is analogous with the training set, such as vqa v2.0 @xcite , are generally favorable. however, latest studies (agrawal et al., 2016; goyal * jinan xu is the corresponding author et al., 2017) have pointed out that many welldeveloped vqa models merely over-exploit the language prior from the training set to provide correct answers without reasoning. that is, the answer prediction might rely more on the correlation to the question and less on the image. for instance, in the vqa-cp v2.0 @xcite training set, the answers of the question with the first few words "how many • • • " are usually "2", and the answers of the specific question "what ′ s the color of the bananas?" are almost all "yellow". consequently, significant drops @xcite are demonstrated while handling with the out-of-distribution test dataset. recently, solutions for this problem can be categorized into two classes, namely, nonaugmentation-based methods @xcite @xcite @xcite @xcite and augmentation-based methods @xcite @xcite . the former seeks to weaken language bias or leverage visual grounding to increase the image dependency, while the latter aims to to balance the dis-tribution of training data. moreover, most of the advanced debiasing methods still suffer from two issues, namely comprehensive bias detecting @xcite and in-distribution (id) generalizability problems @xcite . to solve the above problems, we propose a multi-modal debiasing model with dynamical constraint (mddc). for the first limitation, we construct two bias learning branches. inspired by the way of using the single modality to identify unimodal-specific bias, we adopt a question-only branch for language bias. unfortunately, such a strategy is unsuitable for the bias issue related to visual information. the reason is that the same image is usually used to answer various types of questions, thus the model cannot obtain the specific vision bias that involves necessary information to answer the question, but only an image-to-answer distribution bias. we assume that a more effective way is to provide some question clues for images to generate question-specific vision bias. following this assumption, we design a special bias learning branch by incorporating prompts extracted from questions into an image-only answering model. for the second limitation, we propose a dynamical constraint loss to reduce the difference of the amount of information @xcite between the vqa base module and the bias module. in this way, we dynamically subtract the bias score according to the degree of bias. therefore, we mitigate the problem of uncertain inference caused by the separate optimization of these two modules. we evaluate the proposed mddc on vqa v2.0 and vqa-cp v2.0 benchmarks. experimental results on both datasets demonstrate that our debiasing strategy is competitive compared with mainstream baselines. ## related work visual question answering has witnessed great progress, while a growing body of work @xcite has pointed out the drawbacks of reasoning ability and bias affect. in this section, we review recently proposed vqa debiasing approaches, which can be generally fall into non-augmentation-based methods and augmentation-based methods. ## our approach in this section, we first describe the general architecture of our proposed mddc model and then give the details for each component. figure 3 depicts the overview of our approach, which consists of three major modules: (1) the standard vqa base module, which aims to indicate the probability belonging to each answer candidate; (2) the bias module, which aims to capture biases combining both questions and images simultaneously; (3) the dynamical constraint module, which aims to dynamically control the final prediction distribution. ## conclusion a robust visual question answering model with dynamical constraint is proposed for reducing as much multi-modal bias as possible. compared with previous researches, we investigate a very straightforward way to obtain debiasing effect by subtracting bias score from vqa base score. on one hand, we design a language bias learning branch and a question-guided vision bias learning branch to detect comprehensive biases. on the other hand, a dynamical constraint loss is proposed related to the two bias branches to alleviate the over-correction and insufficient debiasing problems to some extent. experimental results on vqa-cp v2.0 and vqa v2.0 datasets demonstrate the effectiveness of our proposed approach from both quantitative and qualitative perspectives.