Visual Question Answering (VQA ) models tend to rely on the language bias and thus fail to learn the reasoning from visual knowledge , which is however the original intention of VQA . In this paper, we propose a novel cause-effect look at the language bias , where the bias is formulated as the direct  effect of question on answer from the view of causal inference. The effectcan be captured by counterfactual VQA, where the image had not existed in an imagined scenario. Our proposed cause-effect look 1) is general to any baseline VQA architecture , 2) achieves significant improvement on the language-bias sensitive VQA-CP dataset , and 3) fills the theoretical gap in recent language prior based works .
Recent VQA models may tend to rely on language bias as a shortcut and thus fail to sufficiently learn the multi-modal knowledge from both vision and language . In this paper, we investigate how to capture and mitigate language bias in VQA. Motivated by causal effects, we proposed a novel counterfactual inference framework, which enables us to capture the language bias as the direct causal effect of questions on answers and reduce the language bias by subtracting the direct language effect from the total causal effect . Experiments demonstrate that our proposed counterfactual inference framework 1) is general to various VQA backbones and fusion strategies , 2) achieves competitive performance on the language-bias sensitive VQA-CP dataset while performs robustly on the balanced VQA v2 dataset .
R
Visual Question Answering (VQA ) models
Recent VQA models may
R
the language bias
language bias as a shortcut
R
learn the reasoning from visual knowledge , which is however the original intention of VQA
sufficiently learn the multi-modal knowledge from both vision and language
R
propose a novel cause-effect look at
investigate how to capture and mitigate language bias in VQA. Motivated by causal effects, we proposed a novel counterfactual inference framework, which enables us to capture
D
, where the bias is formulated
A
causal effect of questions on answers and reduce the language bias by subtracting the direct language effect from the total causal
R
of question on answer from the view of causal inference. The effectcan be captured by counterfactual VQA, where the image had not existed in an imagined scenario. Our proposed cause-effect look
. Experiments demonstrate that our proposed counterfactual inference framework
R
any baseline VQA architecture
various VQA backbones and fusion strategies
R
significant improvement
competitive performance
R
, and 3) fills the theoretical gap in recent language prior based works
while performs robustly on the balanced VQA v2 dataset