‘Just because you are right, doesn’t mean I am wrong’: Overcoming a bottleneck in development and evaluation of Open-Ended VQA tasks (2021)

Abstract: GQA (CITATION) is a dataset for real-world visual reasoning and compositional question answering. We found that many answers predicted by the best vision-language models on the GQA dataset do not match the ground-truth answer but still are semantically meaningful and correct in the given context. In fact, this is the case with most existing visual question answering (VQA) datasets where they assume only one ground-truth answer for each question. We propose Alternative Answer Sets (AAS) of ground-truth answers to address this limitation, which is created automatically using off-the-shelf NLP tools. We introduce a semantic metric based on AAS and modify top VQA solvers to support multiple plausible answers for a question. We implement this approach on the GQA dataset and show the performance improvements.

## introduction one important style of visual question answering (vqa) task involves open-ended responses such as free-form answers or fill-in-the-blanks. the possibility of multiple correct answers and multi-word responses makes the evaluation of open-ended tasks harder, which has forced vqa datasets to restrict answers to be a single word or a short phrase. despite enforcing these constraints, from our analysis of the gqa dataset (hudson and manning, 2019), we noticed that a significant portion of the visual questions have issues. for example, a question "who is holding the bat?" has only one ground truth answer "batter" while other reasonable answers like "batsman", "hitter" are not credited. we identified six different types of issues with the dataset and illustrated them in a large-scale human-study conducted by @xcite on vqa @xcite and vizwiz @xcite found that almost 50% questions in these datasets have multiple possible answers. datasets had similar observations. the above evidence suggests that it is unfair to penalize models if their predicted answer is correct in a given context but does not match the ground truth answer. with this motivation, we leverage existing knowledge bases and word embeddings to generate alternative answer sets (aas) instead of considering visual questions to have fixed responses. since initially obtained aas are generated from multiple sources and observed to be noisy, we use textual entailment to verify semantic viability of plausible answers to make alternative answer sets more robust. we justify the correctness and quality of the generated aas by human evaluation. we introduce a semantic metric based on aas and train two vision-language models lxmert @xcite and vilbert @xcite on two datasets. the experimental results show that the aas metric evaluates models' performances more reasonably than the old metric. lastly, we incorporate aas in the training phase and show that it further improves on the proposed metric. figure 2 gives an overview of our work. ## related works we discuss related works from two aspects, dataset creation and evaluation. dataset creation-level large-scale vqa datasets are often curated through crowd-sourcing, where open-ended ground-truths are determined by majority voting or annotator agreement. the subjectivity in crowd-sourced datasets is well-studied in human-computer interaction literature- @xcite @xcite etc. @xcite suggested creating a semantically-grounded set of questions issue type definition % ## analysis of gqa dataset gqa is a dataset for real-world visual reasoning and compositional question answering. instead of human annotation, answers to the questions in gqa are generated from the scene graphs of images. we found that automatic creation leads to flaws in the dataset; thus, we manually analyze 600 questions from the testdev balanced split of gqa dataset, and identify six issues shown in figure 1 shows examples of each type of issue. these issues are caused by (not limited to) three reasons. first, the dataset assumes only one ground truth so that other answers with semantic closed meaning are ignored. we propose aas to address this issue to some extent and describe aas in the next section. second, some questions referring to multiple objects cause ambiguous meaning. we leverage scene graphs to address this issue and found 2.92% and 2.94% ambiguous questions in balanced training split and balanced validation split, respectively. these ambiguous questions can be removed from the dataset. third, there are incorrect scene graph detections so that some questions and/or labels do not match with the given images. we plan to address these issues in our future work. ## alternative answer set to credit answers with semantically close meaning as the ground-truth, we propose a workflow that can be visualized from figure 2 . each item in vqa dataset consists @xmath0, q, gt>, where i is an image, q is a question, and gt is a groundtruth answer. we define an alternative answer set (aas) as a collection of phrases {a 1 , a 2 , a 3 ,.., a n } such that a i replaced with gt is still a valid answer to the given image-question pair. we construct aas for each unique ground-truth automatically from two knowledge bases: wordnet @xcite and concpetnet @xcite , two word embeddings: bert @xcite and counter-fitting @xcite . we assign a semantic score to each alternative answer by textural entailment and introduce the aas metric. ## discussion and conclusion to evaluate a model from a semantic point of view, we define an alternative answer set (aas). we develop a workflow to automatically create robust aas for ground truth answers in the dataset using textual entailment. additionally, we did human verification to assess the quality of automatically generated aas. the high agreement score indicates that entailment model is doing a careful job of filtering relevant answers. from experiments on two models and two vqa datasets, we show the effectiveness of aas-based evaluation using our proposed metric. aas can be applied to other tasks, for example, machine translation. bleu @xcite score used to evaluate machine translation models incorporates an average of n-gram precision but does not consider the synonymy. therefore, me-teor @xcite was proposed to overcome this problem. however, meteor only relies on the synset of wordnet to get the synonyms. our proposed aas has the advantage of both knowledge base and word embeddings, which would help better evaluate translation tasks.