E a S e: A Diagnostic Tool for VQA based on Answer Diversity (2021)

Abstract: We propose EASE, a simple diagnostic tool for Visual Question Answering (VQA) which quantifies the difficulty of an image, question sample. EASE is based on the pattern of answers provided by multiple annotators to a given question. In particular, it considers two aspects of the answers: (i) their Entropy; (ii) their Semantic content. First, we prove the validity of our diagnostic to identify samples that are easy/hard for state-of-art VQA models. Second, we show that EASE can be successfully used to select the most-informative samples for training/fine-tuning. Crucially, only information that is readily available in any VQA dataset is used to compute its scores.

## introduction visual question answering (vqa; @xcite requires models to jointly understand an image and a natural language question. this is a challenging task; despite massive training data and recent pre-training strategies @xcite @xcite ) models still struggle to close the gap with oracle performance. vqa datasets (e.g., @xcite consist of image, question pairs for which n human annotators have provided an answer in natural language. when trained on these samples, vqa models are fed with the most frequently chosen answer in the pattern. during inference, the answer with the highest probability is evaluated against the pattern of n ground-truth answers. according to the standard vqa metric @xcite , a model's prediction is considered as perfectly correct if it matches an answer that was frequent in the pattern; less accurate if matching an underrepresented one. this metric implies that, for the majority of cases, several annotators agree on the same exact answer-and a model can thus achieve 100% accuracy in the task. on the other figure 1 : one image from vqa2.0 with two questions and the answers by 10 annotators. frequency of each unique answer (e.g., plaid : 4) and ease values of the samples (the higher, the easier) are reported. in line with recent work aimed at predicting the agreement between annotators (gurari and grauman, 2017), the distribution of answers for a given image, question pair @xcite , or the difficulty of visual questions @xcite , in this paper we introduce ease, a diagnostic tool for vqa which is based on the answers provided to a given question. we propose that two main features of the answer pattern, entropy and semantic content, are informative of the degree of difficulty of a sample. in particular, we conjecture that the more scattered an answer pattern, the more difficult the sample (fig. 1 , down)-unless some or all of those answers are semantically similar (fig. 1 , top). by experimenting with various vqa datasets and models, we first assess the effectiveness of our diagnostic to identify the samples that are easy/difficult for a model. second, we use ease to select increasingly difficult subsets of data that we use to train/fine-tune our models, based on the hypothesis that difficult cases are also more informative during training. in both cases, we show that our simple method is very effective: (1) models are shown to struggle with the most difficult samples according to it; (2) training/fine-tuning models with only a little fraction of samples-the most difficult ones-makes them achieve very high results, which are comparable to models trained/fine-tuned with the whole training data. finally, ease is shown to correlate with the confidence scores provided by human annotators along with their answers, which reveals that it captures a notion of difficulty in line with that by human speakers. ## approach we focus on image, question vqa samples and aim to quantify their difficulty, i.e., how challenging it is for a model to answer them correctly. we propose that the difficulty of a sample can be quantified based on the (readily available) characteristics of the pattern of answers provided by the annotators, and devise a diagnostic tool that builds on this assumption. in particular, we focus on two aspects of the pattern: 1) its entropy, i.e., how scattered it is in terms of the number of unique answer strings; 2) its semantics, i.e., how (dis)similar are the answers in it with respect to their overall semantic representation. we name our diagnostic tool ease and describe it in detail below. ## experiments in hf, we train our vqa models incrementally, first using th samples only, then adding bh samples, and finally using all training samples. the weights for the first stage are initialized randomly; we load the model's weights from previous stages for each incremental stage. for vqa2.0, the percentage of samples for each stage is 9.13% (th), 51.79% (th+bh), and 100% (all), and for vizwiz is 16%, 68.22%, and 100%. we hypothesize that harder splits, i.e., with low ease scores, contain richer multimodal information that could be more informative during a model's learning. for comparison, we also evaluate models in the th(r) condition: we train/fine-tune models with a set of data (with the same size as th) randomly sampled from the training set. we repeat the sampling 10 times, and report the average accuracy. ## results results in tab. 3 support our hypotheses. (1) with only 52% of the training data (th+bh), butd obtains 90% of all validation accuracy (va) in vqa2.0 compared to the model trained on the whole data ( overall, these results indicate that the hard samples selected by ease are more informative than easier ones and help models obtain comparable performance with significantly less training data. ## analysis 6.1 ease vs. question types we explore whether the hard splits selected by ease contain question types that are known to be particularly challenging for vqa models, e.g., count and whquestions. as can be seen in fig. 2 , a higher proportion of wh-(other) and count (number) questions is observed in the hardest split compared to the other splits of vqa2.0. foot_1 in contrast, polar questions (yes/no) are poorly represented in th, which indicates they are overall less challenging for humans and less informative for the models. ## conclusion we present ease, a simple diagnostic tool which quantifies the difficulty of a vqa sample based on its pattern of answers. we show that ease selects the most informative samples of a dataset, which is helpful to train/fine-tune vqa models more efficiently with less, but highly-informative data. in future work, we plan to combine model prediction for difficulty estimation in ease.