Overcoming Language Priors with Counterfactual Inference for Visual Question Answering (2023)

Abstract: “Recent years have seen a lot of efforts in attacking the issue of language priors in the field ofVisual Question Answering (VQA). Among the extensive efforts, causal inference is regarded asa promising direction to mitigate language bias by weakening the direct causal effect of questionson answers. In this paper, we follow the same direction and attack the issue of language priorsby incorporating counterfactual data. Moreover, we propose a two-stage training strategy whichis deemed to make better use of counterfactual data. Experiments on the widely used bench-mark VQA-CP v2 demonstrate the effectiveness of the proposed approach, which improves thebaseline by 21.21% and outperforms most of the previous systems.”

## introduction as an ai-complete task to answer questions about visual content, visual question answering (vqa) has seen surging interest in recent years. the task is thought to be extremely challenging since a vqa system requires the capability of visual and language understanding and the capability of multi-modal reasoning. recent researches in this field have paid increasing attention to the issue of language priors, aka language bias @xcite . the issue of language priors is caused by spurious correlation between the question pattern and the answer. see the example in figure 1 , "yellow" is the most likely answer to the question "what color are the bananas" in the training data. so a simple solution to answering the question is to give the answer "yellow" with no reference to visual content. such a short cut can achieve an accuracy of 54.5% for the question. to overcome language priors in vqa, previous works generally resort to data augmentation. in this direction, visual and textual explanations can be used as the data for augmentation @xcite . besides, counterfactual training samples are also regarded as a valuable source for the purpose @xcite @xcite . in the direction of causal effect for vqa, more recent work is counterfactual vqa that focuses on the inference instead of training phase @xcite , though, we still think of counterfactual data augmentation as an efficient and effective way to solve the issue of language priors. so in this paper we first design novel causal graphs specifically for the task of vqa, and then use the causal graphs to guide the generation of counterfactual data. finally, to make better use of counterfactual data, we propose a two-stage training strategy. we evaluate the proposed approach on the widely used benchmark vqa-cp v2. extensive experiments demonstrate the effectiveness of the approach, which improves over the baseline by 21.21% and outperforms most of previous systems. moreover, to evaluate the generalization ability of the approach, we also experiment with vqa v2 and find that our approach achieves the best performance on the dataset. the contributions of the paper are as follows. ## methodology in this section, we first describe the implementation of our baseline system. then we introduce the design of vqa causal graphs which inspire us to come up with the proposed methods. finally we describe the methods in detail. the system framework is presented in figure 2 . ## conclusion and future work to mitigate the effect of language priors in the vqa task, we proposed a causal inference approach that automatically generates counterfactual data and utilize the data in a two-stage training strategy. we also designed several causal graphs to guide the generation of counterfactual data. extensive experiments on the benchmark vqa-cp v2 shows that our system achieves significant improvements over the baselines and outperforms most of previous works. moreover, our system achieves the best performance on vqa v2 which demonstrates the capability of generalization. the starting point of the the second stage training is critical to the performance, in our future work, we would like to determine the starting point in an automatic way. moreover, it is interesting to evaluate the performance when other networks such as san are used as the backbone. we will also study this problem in our future work.