Digging out Discrimination Information from Generated Samples for Robust Visual Question Answering (2023)

Abstract: Visual Question Answering (VQA) aims to answer a textual question based on a given image. Nevertheless, recent studies have shown that VQA models tend to capture the biases to answer the question, instead of using the reasoning ability, resulting in poor generalisation ability. To alleviate the issue, some existing methods consider the natural distribution of the data, and construct samples to balance the dataset, achieving remarkable performance. However, these methods may encounter some limitations: 1) rely on additional annotations, 2) the generated samples may be inaccurate, e.g., assigned wrong answers, and 3) ignore the power of positive samples. In this paper, we propose a method to Dig out Discrimination information from Generated samples (DDG) to address the above limitations. Specifically, we first construct positive and negative samples in vision and language modalities, without using additional annotations. Then, we introduce a knowledge distillation mechanism to promote the learning of the original samples by the positive samples. Moreover, we impel the VQA models to focus on vision and language modalities using the negative samples. Experimental results on the VQA-CP v2 and VQA v2 datasets show the effectiveness of our DDG.

## introduction with the vigorous development of computer vision and natural language processing fields, it has promoted the vision-and-language @xcite field to take a forward step. as a typical task of the vision-and-language field, visual question answering (vqa) @xcite requires an agent to fully comprehend the information of the questions and images, and then correctly answer the textual question according to the image. although recent advances @xcite have achieved impressive performance on the benchmark datasets * corresponding author (e.g., vqa v2 @xcite ), numerous studies @xcite have shown that some vqa models tend to excessively rely on the superficial correlations (i.e., biases) between the questions and answers, instead of adopting reasoning ability to answer the questions. for example, the vqa models can easily answer "2" and "tennis" for the questions "how many . . . " and "what sports . . . ", respectively, would obtain higher accuracy, since the corresponding answers "2" and "tennis" are occupied the most in the dataset. however, memorising the biases to answer the questions would signify flawed reasoning ability, resulting in poor generalisation ability. to mitigate the bias issues, many methods have been proposed, which can be roughly categorised into three types: 1) enhance visual attention @xcite , 2) directly weaken the biases @xcite , and 3) balance the dataset @xcite . previous studies have shown the methods that balance the dataset usually outperform other types of methods, since they dig out the natural distribution of the data, and then devise a suitable strategy to overcome the biases. specifically, css @xcite and mutant @xcite ) methods generate counterfactual samples by masking the critical objects or words in the images and questions, respectively. however, these methods require additional annotations that are hard to obtain. to get rid of the dependence on the additional annotations, mmbs @xcite constructs the positive questions by randomly shuffling the question words or removing the words of question types, which destroys the grammar and semantics of the original questions. moreover, simpleaug @xcite and kd-daug @xcite build the new samples by re-combining the existing questions and images, which may be difficult to assign correct answers for the generated samples. ssl-vqa @xcite and d-vqa @xcite construct the negative samples by randomly sampling the images or questions in a mini-batch data. nevertheless, these methods consider the negative samples only, but ignoring the generated positive samples would improve the diversity of the dataset and further promote the robustness of the vqa models. to overcome the above issues, we propose a method to dig out discrimination information from generated samples (ddg). as pointed out by @xcite , the bias issues exist in both vision and language modalities, we thus construct positive and negative samples in vision and language modalities and devise corresponding training objectives to achieve unbiased learning. concretely, we feed the samples to the updn @xcite ) model pre-trained on the vqa-cp @xcite v2 training set, and select k objects based on the top-k image attention weights of the updn as the positive images. the positive questions can be constructed by using the translate-and-back-translate mechanism, e.g., @xmath0. we then combine the positive images and positive questions with original questions and original images, respectively, as positive image and question samples. based on the positive samples, we adopt a knowledge distillation mechanism @xcite to help the learning of the original samples. moreover, inspired by @xcite , we construct mismatched image-question pairs as negative samples. generally speaking, one cannot answer the question correctly given the mismatched image-question pairs, since missing the supporting modality information. to promote the vqa models to focus on the vision and language modalities, we devise a training objective that aims to minimise the likelihood of predicting the ground-truth answers of the original samples when given the corresponding negative samples. besides, we further introduce the corresponding positive samples to assist the training. based on the above debiased techniques, our ddg achieves impressive performance on the vqa-cp v2 @xcite and vqa v2 @xcite datasets, which demonstrates the effectiveness of our ddg. our contributions can be summarised as follows: 1) we devise a novel positive image samples generation strategy that uses the image attention weights of the pre-trained updn model to guide the selection of the target objects. 2) we introduce the knowledge distillation mechanism to promote the learning of the original samples by the positive samples. 3) we adopt the positive and negative samples to impel the vqa models to focus on the vision and language modalities, to mitigate the biases. ## digging out discrimination information from generated samples as shown by @xcite , vqa models tend to capture the biases in a dataset to answer questions, instead of adopting the reasoning ability, resulting in poor generalisation ability. moreover, bias issues exist in both vision and language modalities @xcite . to address the above issues, we seek to construct both positive and negative samples in vision and language modalities, and devise corresponding debiased strategies to achieve unbiased learning. the overall framework is shown in figure . 1. ## conclusion in this paper, we have proposed a novel method named ddg to alleviate the bias issues in vqa from vision and language modalities. specifically, we construct both positive and negative samples in vision and language modalities without using additional annotations, in which the positive questions have similar semantics to the original questions, while the positive images contain foreground information. based on the positive samples, we heuristically introduce the knowledge distillation mechanism to facilitate the training of the original samples through guidance from positive samples. moreover, we put forth a strategy that encourages vqa models to focus more on the vision and language modalities when answering the questions, aided by the negative samples. extensive experiments on the vqa-cp v2 and vqa v2 datasets show the effectiveness of our ddg.