Check It Again:Progressive Visual Question Answering via Visual Entailment (2021)

Abstract: While sophisticated neural-based models have achieved remarkable success in Visual Question Answering (VQA), these models tend to answer questions only according to superficial correlations between question and answer. Several recent approaches have been developed to address this language priors problem. However, most of them predict the correct answer according to one best output without checking the authenticity of answers. Besides, they only explore the interaction between image and question, ignoring the semantics of candidate answers. In this paper, we propose a select-and-rerank (SAR) progressive framework based on Visual Entailment. Specifically, we first select the candidate answers relevant to the question or the image, then we rerank the candidate answers by a visual entailment task, which verifies whether the image semantically entails the synthetic statement of the question and each candidate answer. Experimental results show the effectiveness of our proposed framework, which establishes a new state-of-the-art accuracy on VQA-CP v2 with a 7.55% improvement.

## introduction visual question answering (vqa) task is a multimodal problem which requires the comprehensive understanding of both visual and textual information. presented with an input image and a question, the vqa system tries to determine the correct answer in the large prediction space. recently, some studies @xcite @xcite demonstrate that vqa systems suffer from the superficial correlation bias (i.e. language priors) caused by accidental correlations between answers and questions. as a result, traditional vqa models always output the most common answer(selvaraju et al., 2019) of the input sample's question category, no matter what image is given. to address this language priors problem, various approaches have been developed. however, through exploring the characteristics of the existing methods, we find that whether the general vqa models such as updn @xcite and lxmert @xcite or models carefully designed for language priors, as lmh @xcite and ssl @xcite , yield a non-negligible problem. both models predict the correct answer according to one best output without checking the authenticity of answers. besides, these models have not made good use of the semantics information of answers that could be helpful for alleviating the language-priors. as presented in figure 1 (a), quite a few correct answers usually occur at top n candidates rather than top one. meanwhile, if the top n candidate answers are given, the image can further verify the visual presence/absence of concepts based on the combination of the question and the candidate answer. as shown in figure 1 (b), the question is about the color of the bat and two candidate answers are "yellow" and "black". after checking the correctness of candidate answers, the wrong answer "yellow" which is contradicted with the image can be excluded and the correct answer "black" which is consistent with the image is confirmed. nevertheless, this visual verification, which utilizes answer semantics to alleviate language priors, has not been fully investigated. in this paper, we propose a select-and-rerank (sar) progressive framework based on visual entailment. the intuition behind the proposed framework comes from two observations. first, after excluding the answers unrelated to the question and image, the prediction space is shrunken and we can obtain a small number of candidate answers. second, on the condition that a question and one of its candidate answer is bridged into a complete statement, the authenticity of this statement can be inferred by the content of the image. therefore, after selecting several possible answers as candidates, we can utilize the visual entailment, consisting of image-text pairs, to verify whether the image semantically entails the synthetic statement. based on the entailment degree, we can further rerank candidate answers and give the model another chance to find the right answer. to summarize, our contributions are as follows: ## related work language-priors methods to address the language prior problem of vqa models, a lot of approaches have been proposed, which can be roughly categorized into two lines: (1) design-ing specific debiasing models to reduce biases. most works of this line are ensemble-based methods @xcite @xcite @xcite , among these, lmh @xcite reduces all biases between question-answer pairs by penalizing the samples that can be answered without utilizing image content. (2) data augmentation to reduce biases. the main idea of such works @xcite @xcite is to carefully construct more balanced datasets to overcome priors. for example, the recent method ssl @xcite first automatically generates a set of balanced question-image pairs, then introduces an auxiliary self-supervised task to use the balanced data. css @xcite balances the data by adding more complementary samples which are generated by masking objects in the image or some keywords in the question. based on css, cl @xcite forces the model to utilize the relationship between complementary samples and original samples. unlike ssl and css which do not use any extra manual annotations, mutant @xcite locates critical objects in the image and critical words in the question by utilizing the extra object-name labels, which directly helps the model to ground the textual concepts in the image. however, above methods only explore the interaction between the image and the question, ignoring the semantics of candidate answers. in this paper, we propose a progressive vqa framework sar which achieves better interaction among the question, the image and the answer. answer re-ranking although answer reranking is still in the infancy in vqa task, it has been widely studied for qa tasks like open-domain question answering, in which models need to answer questions based on a broad range of opendomains knowledge sources. recent works @xcite address this task in a two-stage manner: extract candidates from all passages, then focus on these candidate answers and rerank them to get a final answer. rankvqa @xcite introduces answer re-ranking method to vqa task. they design an auxiliary task which reranks candidate answers according to their matching degrees with the input image and off-line generated image captions. however, rankvqa still predicts the final answer from the huge prediction space rather than selected candidate answers. ## method figure 2 shows an overview of the proposed selectand-rerank (sar) framework, which consists of a candidate answer selecting module and an answer re-ranking module. in the candidate answer selecting module, given an image and a question, we first use a current vqa model to get a candidate answer set consisting of top n answers. in this module, the answers irrelevant to the question can be filtered out. next, we formulate the vqa as a ve task in the answer re-ranking module, where the image is premise and the synthetic dense caption @xcite (combination of the answer and the question ) is hypothesis. we use the cross-domain pre-trained model lxmert @xcite as ve scorer to compute the entailment score of each image-caption pair, and thus the answer corresponding to the dense caption with the highest score is our final prediction. ## conclusion in this paper, we propose a select-and-rerank (sar) progressive framework based on visual entailment. specifically, we first select candidate answers to shrink the prediction space, then we rerank candidate answers by a visual entailment task which verifies whether the image semantically entails the synthetic statement of the question and each candidate answer. our framework can make full use of the interactive information of image, question and candidate answers. in addition, it is a generic framework, which can be easily combined with the existing vqa models and further boost their abilities. we demonstrate advantages of our framework on the vqa-cp v2 dataset with extensive experiments and analyses. our method establishes a new state-of-the-art accuracy of 66.73% with an improvement of 7.55% on the previous best.