Overcoming Language Priors in Visual Question Answering via Distinguishing Superficially Similar Instances (2022)

Abstract: Despite the great progress of Visual Question Answering (VQA), current VQA models heavily rely on the superficial correlation between the question type and its corresponding frequent answers (i.e., language priors) to make predictions, without really understanding the input. In this work, we define the training instances with the same question type but different answers as superficially similar instances, and attribute the language priors to the confusion of VQA model on such instances. To solve this problem, we propose a novel training framework that explicitly encourages the VQA model to distinguish between the superficially similar instances. Specifically, for each training instance, we first construct a set that contains its superficially similar counterparts. Then we exploit the proposed distinguishing module to increase the distance between the instance and its counterparts in the answer space. In this way, the VQA model is forced to further focus on the other parts of the input beyond the question type, which helps to overcome the language priors. Experimental results show that our method achieves the state-of-the-art performance on VQA-CP v2. Codes are available at Distinguishing-VQA.

## introduction recent years have witnessed great progress in vqa based on deep learning. however, some researchers reveal that most existing vqa models heavily rely on the superficial correlation between the question type and its corresponding frequent answers to make predictions, instead of really understanding the input @xcite @xcite . for example, once a vqa model detects that a question begins with "how many", it tends to blindly output the most common answer "2" to the "how many" questions in the training data without looking at other parts of the input. (b) visualizes the projection of such instances in the answer space obtained by san @xcite and updn @xcite respectively. we observe that both models could not distinguish well between the mixed superficially similar instances. we argue that the language priors arise from that the vqa model only captures the question type and ignores other information in the input imagequestion pair. on the one hand, the datasets which they are built upon are usually biased. for the questions of the same question type, the distribution of answers is severely biased to some frequent answers. on the other hand, deep learning methods tend to memorize some simple and salient patterns (e.g., frequency) in the training data, and easily exploit a shortcut to make predictions. therefore, given an instance for testing, they prefer to directly look at the question type and leverage the superficial correlation rather than also analyze other information and further understand the whole input. for solving the above problem, we introduce the concept of superficially similar instances, and propose to overcome the language priors via distinguishing between such instances. as shown in figure 1a , the superficially similar instances refer to the instances that have the same question type but different answers. by distinguishing such instances, the vqa model is forced to focus on other information besides the question type in the input image-question pair when making predictions. unfortunately, it is not trivial to perform the distinguishing task, since a classical vqa paradigm does not have such a mechanism to support this. given a training instance, the paradigm usually adopts the (binary) cross-entropy loss to reduce the distance between the projection of image-question pair in the answer space and the ground-truth, without explicitly encouraging the differentiation of superficially similar instances. as shown in figure 1b , the superficially similar instances are mixed up in the answer space, which indicates that both two widely-used vqa models @xcite could not distinguish between them well. in this work, we propose a novel training framework that explicitly encourages the vqa model to distinguish between the superficially similar instances. specifically, for each training instance, we first construct a superficially similar set. the set consists of two kinds of superficially similar counterparts for the instance, which are collected in different ways and complement each other. then we exploit the designed distinguishing module to increase the distance between a training instance and its superficially similar counterparts in the answer space. given a training instance, the distinguishing module urges the vqa model to give a higher probability on its ground-truth answer to itself than its superficially similar counterparts. finally, considering the cost of time and space, we implement our method in a resource-efficient way by manipulating the high-level features to construct the superficially similar set and sampling from the set. in summary, the main contributions of this paper are as follows: our method achieves the state-of-the-art results on the benchmark dataset vqa-cp v2, while maintaining competitive performance on the standard vqa v2 dataset. ## related works despite the great progress in visual question answering @xcite , some researchers observe that most existing vqa models heavily rely on the language priors to make decisions @xcite @xcite . recently, @xcite propose a new split of the vqa v1 and vqa v2 datasets (vqa-cp v1 and vqa-cp v2 respectively), which makes the answer distribution of each question type different between the train and test splits. they find that the performance of existing vqa models drops significantly on their new splits compared to the original splits. this fully demonstrates the necessity of overcoming the language priors in vqa. previous works on overcoming the language priors in vqa, which is called debiasing vqa methods, can be roughly divided into four categories. methods modifying model architecture. they usually design a specific model architecture to decompose the process of vqa into several steps. for instance, @xcite propose a grounded visual question answering (gvqa) model to disentangle the recognition of visual concepts from the identification of plausible answers. similarly, jing et al. (2020 ## methodology the whole framework of our method is depicted in figure 2 . in the training phase, we optimize the whole framework with the loss l vqa of a classical vqa paradigm and the proposed distinguishing loss l dis in a multi-task manner, and the total loss l is defined as follows: given a training instance, we compute its distinguishing loss l dis in two steps. first, we offline construct a superficially similar set for the instance, which consists of its superficially similar counterparts. second, when online training, we input the given instance with its superficially similar set into the vqa model, and further exploit the distinguishing module to compute l dis . ## conclusion in this paper, we introduce the concept superficially similar instances and propose a novel training framework for overcoming language priors by explicitly encouraging the vqa model to distinguish between such instances. extensive experiments show that our approach achieves the state-of-the-art results on vqa-cp v2, while maintaining competitive performance on vqa v2. qualitative analysis also demonstrates that our method alleviates the language priors effectively and really understands the input.