Learning to Contrast the Counterfactual Samples for Robust Visual Question Answering (2020)

Abstract: In the task of Visual Question Answering (VQA), most state-of-the-art models tend to learn spurious correlations in the training set and achieve poor performance in out-of-distribution test data. Some methods of generating counterfactual samples have been proposed to alleviate this problem. However, the counterfactual samples generated by most previous methods are simply added to the training data for augmentation and are not fully utilized. Therefore, we introduce a novel self-supervised contrastive learning mechanism to learn the relationship between original samples, factual samples and counterfactual samples. With the better cross-modal joint embeddings learned from the auxiliary training objective, the reasoning capability and robustness of the VQA model are boosted significantly. We evaluate the effectiveness of our method by surpassing current state-of-the-art models on the VQA-CP dataset, a diagnostic benchmark for assessing the VQA model’s robustness.

## introduction to develop human-like visual and language understanding of ai, the task of answering a question about the given visual content has been proposed, i.e., visual question answering (vqa) @xcite . although the current state-of-the-art methods @xcite @xcite can achieve good results on the vqa benchmarks such as vqa v2 @xcite , recent researches @xcite @xcite have found that these methods tend to explore superficial correlations in the training set and perform poorly when transferred to real world setting. specifically, given a question "what color is the banana?", the models prefer to take the shortcut and "assume" that the answer should be "yellow" since it is the most common answer in the training set, rather is this man holding a cat? 𝑸 is this man holding a [mask]? is [mask] [mask] [mask] [mask] cat? 𝑰 yes. yes. not yes. 𝒂 𝑰 + 𝑰 - 𝑸 + 𝑸 - 𝒂 + 𝒂 - @xcite . a successful robust and unbiased vqa system is supposed to be able to deduce the right answer from the right area of the image. lately, some studies have proposed to synthesize counterfactual samples to improve the robustness of vqa models. @xcite apply gan @xcite to generate images. css algorithm proposed by @xcite generates counterfactual samples by masking the critical objects in images or words in questions, as shown in figure 1 . the critical objects or words can be obtained from css as by-products. nevertheless, the counterfactual samples are simply added to the training data for augmentation, ignoring that the relationship between original samples and counterfactual samples are vital for the reasoning of vqa models. specifically, the model should be able to learn why the correct answer cannot be inferred after changing the original sample to the counterfactual sample. we posit that modeling the relationship between original samples, factual samples and counterfactual samples can bring more self-supervised signals to improve the reasoning ability of the model. in order to enable the vqa model to understand the impact of the samples changing from original to counterfactual, we introduce a novel contrastive learning mechanism into the training with counterfactual samples, which is first proposed in the field of learning with counterfactual samples. the auxiliary contrastive training objective model the relationship between original samples, factual samples and counterfactual samples in the cross-modal joint embedding space. with the better cross-modal representations, both the reasoning ability and robustness of the vqa model are improved efficiently. overall, the contributions of this paper are as follows: ## methodology in this section, we introduce our technical realization. the flowchart of our proposed method is illustrated in figure 2 . our method consists of three parts: (1) a base vqa model (2) a factual and counterfactual samples synthesizing (css) module (3) a contrastive learning (cl) objective. ## conclusion in order to fully utilize the supervision information of synthesized counterfactual samples in robust vqa, we introduce a self-supervised contrastive learning mechanism to learn the relationship between factual samples and counterfactual samples. the experimental results demonstrate that our method improves the reasoning ability and robustness of the vqa models.