Towards Robust Visual Question Answering: Making the Most of Biased Samples via Contrastive Learning (2022)

Abstract: Models for Visual Question Answering (VQA) often rely on the spurious correlations, i.e., the language priors, that appear in the biased samples of training set, which make them brittle against the out-of-distribution (OOD) test data. Recent methods have achieved promising progress in overcoming this problem by reducing the impact of biased samples on model training. However, these models reveal a trade-off that the improvements on OOD data severely sacrifice the performance on the in-distribution (ID) data (which is dominated by the biased samples). Therefore, we propose a novel contrastive learning approach, MMBS, for building robust VQA models by Making the Most of Biased Samples. Specifically, we construct positive samples for contrastive learning by eliminating the information related to spurious correlation from the original training samples and explore several strategies to use the constructed positive samples for training. Instead of undermining the importance of biased samples in model training, our approach precisely exploits the biased samples for unbiased information that contributes to reasoning. The proposed method is compatible with various VQA backbones. We validate our contributions by achieving competitive performance on the OOD dataset VQA-CP v2 while preserving robust performance on the ID dataset VQA v2.

## introduction visual question answering (vqa), aiming to answer a question about the given image, is a multimodal task that involves the intersection between vision and language. despite the remarkable performance on many vqa datasets such as vqa v2 @xcite , recent studies @xcite @xcite figure 1 : qualitative comparison of our method lmh+mmbs against the plain method updn and the debiasing method lmh. in vqa-cp v2 (upper), the question types ('does the' and 'how many') bias updn to the most common answers (see fig. 5 for the answer distribution). lmh alleviates the language priors for yesno questions (upper left), while it fails on the more difficult non-yesno questions (upper right). besides, lmh damages the id performance, giving an uncommon answer to the common sample from vqa v2 (lower left). mmbs improves the ood performance while maintains the id performance (lower right). recently, several methods achieved remarkable progress in overcoming this language prior problem. they assign less importance to the biased samples that can be correctly classified with the spurious correlation. however, most of them achieve gains on vqa-cp v2 at the cost of degrading the model's id performance on the vqa v2 dataset (see tab. 2 ). this trade-off suggests that the success of these methods merely comes from biasing the models to other directions, rather than endow-ing them with the reasoning capability and robustness to language priors. ideally, a robust vqa system should maintain its performance on the id dataset while overcoming the language priors, as shown in fig. 1 . we think the essence of both language-prior and trade-off problems is about the learning of biased samples. the former is caused by over-reliance on biased information from biased samples, while the latter is caused by undermining the importance of biased samples. therefore, if a model can precisely exploit the biased samples for intrinsic information of the given task, both problems can be alleviated simultaneously. motivated by this, we propose a self-supervised contrastive learning method (mmbs) for building robust vqa systems by make the most of biased samples. firstly, in view of the characteristics of the spurious correlations, we construct two kinds of positive samples for the questions of training samples to exploit the unbiased information, and then design four strategies to use the constructed positive samples. next, we propose a novel algorithm to distinguish between biased and unbiased samples, so as to treat them differently. on this basis, we introduce an auxiliary contrastive training objective, which helps the model learn a more general representation with ameliorated language priors by narrowing the distance between original samples and positive samples in the cross-modality joint embedding space. to summarize, our contributions are as follow: i) we propose a novel contrastive learning method, which effectively addresses the language prior problem and the id-ood performance trade-off in vqa, by making the most of biased samples. ii) we propose an algorithm to distinguish between biased and unbiased samples and treat them differently in contrastive learning. iii) experimental results demonstrate that our method is compatible with various vqa backbones and achieve competitive performance on the language-bias sensitive vqa-cp v2 dataset while preserving the original accuracy on the in-distribution vqa v2 dataset. ## related work overcoming language priors in vqa. recently, the language biases in vqa datasets raised the attention of many researchers @xcite @xcite . in response to this problem, numerous methods are proposed to debias the vqa models. the most effective ones of them can be roughly divided into two categories: ensemblebased methods @xcite @xcite @xcite introduce a biased model, which is designed to focus on the spurious features, to assist the training of the main model. for example, the recent method lpf @xcite leverages the output distribution of the bias model to down-weight the biased sample when computing the vqa loss. however, these methods neglect the useful information that helps reasoning in biased samples. databalancing methods @xcite balance the training priors. for example, css and mutant @xcite generate samples by masking the critical object in images and word in questions and by semantic image mutations respectively. these methods usually outperform other debiasing methods with a large margin on vqa-cp v2, because they bypass the challenge of the imbalanced settings @xcite by explicitly balancing the answers' distribution at the training stage. though our method constructs the positive questions, it does not change the training answers' distribution. we also extend our method to the data-balancing method sar @xcite . contrastive learning in vqa. recently, the contrastive learning is well-developed in unsupervised learning @xcite while its application in vqa is still in initial stage. cl @xcite is the first work to employ contrastive learning to improve vqa model's robustness. its motivation is to learn a better relationship among the input sample and the factual and counterfactual sample which are generated by css. however, cl brings weak ood performance gain and id performance drop based on css. in contrast, our method attributes the key point of solving language bias to the positive-sample designs for excluding the spurious correlations. it is modelagnostic and can boost models' ood performance significantly while retain the id performance. ## conclusion in this paper, we propose a novel contrastive learning method to ameliorate the id-ood trade-off problem faced by most existing debaising methods for vqa models. instead of undermining the importance of the biased samples, our method makes the most of them via contrastive learning. considering the characteristics of language priors, we design the positive samples which eliminate the biased information. on this basis, we investigate several strategies to use the positive samples and design an algorithm that treat biased and unbiased samples differently in contrastive learning. the proposal is compatible with multiple backbone models and debiasing methods, and achieves competitive performance on ood vqa-cp v2 while maintaining the performance on id vqa v2. meanwhile, our approach provides insights on how to avert the trade-off between in-distribution and out-ofdistribution performance.