A Robust Dual-debiasing VQA Model based on Counterfactual Causal Effect (2024)

Abstract: Traditional VQA models are inherently vulnerable to language bias, resulting in a significant performance drop when encountering out-of-distribution datasets. The conventional VQA models suffer from language bias that indicates a spurious correlation between textual questions and answers. Given the outstanding effectiveness of counterfactual causal inference in eliminating bias, we propose a model agnostic dual-debiasing framework based on Counterfactual Causal Effect (DCCE), which explicitly models two types of language bias(i.e., shortcut and distribution bias) by separate branches under the counterfactual inference framework. The effects of both types ofbias on answer prediction can be effectively mitigated by subtracting direct effect of textual questions on answers from total effect ofvisual questions on answers. Experimental results demonstrate that our proposed DCCE framework significantly reduces language biasand achieves state-of-the-art performance on the benchmark datasets without requiring additional augmented data. Our code is available inhttps://github.com/sxycyck/dcce.

## introduction visual question answering (vqa) @xcite is a challenging task that seeks to develop intelligent systems capable of generating answers to questions about a given image. recently, deep neural networks (dnn) have demonstrated robust representation learning capabilities across different types of modality data and have produced impressive results on vqa tasks @xcite @xcite . however, most existing vqa models only perform well on in-distribution (id) datasets, but still are vulnerable with out-of-distribution (ood) datasets where the answer distributions for the same question type in particular, as shown in fig. 1 (a), the distribution of answers for some question types is imbalanced. for example, in the vqa-cp validation set, simply answering "yes" to the questions "do you..." can achieve 95% accuracy, whereas answering "1" to the questions "how many..." can achieve 60% accuracy. that is, they tend to directly output answers based on the answer distribution for certain question type , resulting in distribution bias. in addition, most dnn-based vqa methods exhibit over-reliance on datasets, which would induce dnns to learn spurious correlations between questions and answers, resulting in shortcut bias. this bias arises from question-answer shortcut rather than proper visual grounding. @xcite , such as the frequent co-occurrence of question elements (e.g., keywords) and answers. as shown in answer "right" when both words "hand" and "holding" appear in the question by shortcut correlation. this shortcut bias can result in a significant performance drop when the test set exhibits different statistical regularities. mitigating the aforementioned language bias has been crucial for enhancing vqa performance. current debiasing methods for vqa often address language bias by incorporating an extra question-only branch @xcite . initially, this extra branch is employed to capture language bias from the training data, and subsequently, it is removed to mitigate the bias's effect on answer predictions. however, the rough branch-excluding operation cannot efficiently disentangle the "good" language context from the captured language bias @xcite . inspired by counterfactual reasoning and causal effects @xcite , the recent cf-vqa @xcite addresses this issue by formulating language bias as the direct effect of questions on answers through the question-only branch, and mitigates the bias by subtracting the direct language effect from the total causal effect of visual questions on answers. although cf-vqa shows encouraging results, it addresses language bias with a single branch. this is inadequate to capture both shortcut bias and distribution bias that represents two complementary aspects of the inherent language bias, which, if confused, would hamper performance @xcite . to address the above issues, we propose a modelagnostic dual-debiasing framework based on counterfactual causal effect (dcce) @xcite , which can effectively mitigate the shortcut bias and distribution bias, each representing distinct facets of language bias. specifically, dcce formulates the shortcut bias and the distribution bias as a direct causal effect of the content of the questions and the types of questions on the answers, respectively. the above two types of bias are mitigated by subtracting the corresponding direct causal effect from the total causal effect, which represents the causal effect of visual questions on answers. for our method, whether facing id or ood datasets, the prediction is not affected by data distribution interference, so it is robust. to distinguish the effect of each bias type and multimodal reasoning in vqa, as shown in fig. 2 , during the training phase, dcce develops two ensemble models that combine the vqa model with each of the two bias branches. a questiononly branch is used to learn the shortcut bias, while a question type-only branch is used to learn the distribution bias. in this case, dcce can easily mitigate the effects of two biases for the testing by subtracting the respective direct effects from the total effect. this is achieved by subtracting the logit sums of the question-only and question type-only branches from the total logit sum. note that our dcce is model-agnostic and can integrate different baseline vqa models. in summary, the contributions of our work can be summarized as follows. ## conclusion this research introduces a novel vqa model designed to mitigate shortcut bias and distribution bias in vqa tasks by leveraging counterfactual causal effects. furthermore, we explore how various configurations can mitigate language bias in vqa within the framework of counterfactual inference. experimental results show show that answers obtained through the causal effect tie exhibit superior accuracy, highlighting the efficacy of the method.