Understanding ME ? Multimodal Evaluation for Fine-grained Visual Commonsense (2022)

Abstract: Visual commonsense understanding requires Vision Language (VL) models to not only understand image and text but also cross-reference in-between to fully integrate and achieve comprehension of the visual scene described. Recently, various approaches have been developed and have achieved high performance on visual commonsense benchmarks. However, it is unclear whether the models really understand the visual scene and underlying commonsense knowledge due to limited evaluation data resources. To provide an in-depth analysis, we present a Multimodal Evaluation (ME) pipeline to automatically generate question-answer pairs to test models’ understanding of the visual scene, text, and related knowledge. We then take a step further to show that training with the ME data boosts the model’s performance in standard VCR evaluation. Lastly, our in-depth analysis and comparison reveal interesting findings: (1) semantically low-level information can assist the learning of high-level information but not the opposite; (2) visual information is generally under utilization compared with text.

## introduction vision language (vl) understanding is challenging because it requires vl models to identify and integrate information from both modalities to fully understand visual scenes. numerous vl benchmarks have been created such as clever @xcite , gqa (hudson and manning, 2019), vqa @xcite , vcr @xcite and snli-ve @xcite . these benchmarks typically form vl evaluation in question-answering format with images and test models' understanding of both vl modalities. despite the high accuracy achieved by existing large pretrained vl models, recent works have pointed out that vl models tend to exploit data biases such figure 1 : an example from vl benchmark, visual commonsense reasoning (vcr) @xcite . vl models can answer the highly semantic vcr question correctly but fail terribly in answering related visual question (q2), textual question (q3), and background knowledge question (q4). as shallow mappings with language priors and unbalanced utilization of information between modalities @xcite @xcite . as in fig. 1 , notwithstanding the model's success in answering q1, the same model fails to answer the related visual, textual, and background questions. this example demonstrates that the vl model does not fully understand the visual scene, which leads to prediction inconsistency (when one model makes conflicting(inconsistent) predictions in two related questions). in our analysis, prediction inconsistency is surprisingly common among models in different modalities. former works have also pointed out that most vqa systems achieve only middling self-consistency (60 -80%) @xcite . therefore, we pose doubts to existing vl models' ability to thoroughly comprehend visual commonsense despite their high accuracy performance on the leaderboards. in this work, we propose to evaluate models' understanding and consistency in predictions across modalities. for that intention, we propose a multimodal evaluation (me) evaluation schema that can augment existing vl benchmarks like vcr. for any given sample of the vl data, e.g. imagequestion-answer pair, me first retrieves and extracts related information of three modalities: vision, text, and background knowledge. after that, it unifies the information across modalities via a multimodal graph and further automatically generates related sub-questions corresponding to all three modalities (as examples shown in fig. 1 ). the sub-questions would be semantically relevant to the input image-question-answer pair, and therefore, after answering the original input question, we can further utilize them to evaluate existing vl models' understanding across the three modalities and pinpoint their shortcoming and biases. under minimal human verification, with me, we create multimodal sub-question evaluation benchmark with 630k multiple choice sub-questions for 110k images from vcr @xcite : 110k of them are visual; 260k of them are about text; and the rest 260k are related to background knowledge. after in-depth evaluation and analysis with topperforming vl models, we discover a few interesting findings: (1) semantically low-level information can assist learning of high-level information but not the opposite; (2) visual information is generally under utilization compared with text. (3) vl models may struggle to utilize related background knowledge information. besides, we propose a multimodal coaching (mc) framework to conditionally augment subquestions in training. depending on vl models' behavior, mc would conditionally decide if it should augment to reinforce the understanding of a particular modality. we show that by using mc, we not only improve models' consistency but also the overall performance. for example, mc boosts the performance of vl-bert by more than 1% on the original vcr q2a metric and even more then 7% in sub-question evaluation metric. our contributions include: ## related work biases occur if vl models cannot comprehensively understand the contents of both images and texts. they need to not only understand information from the two modalities respectively but also integrate these information by cross-referencing. @xcite pointed out biases like unbalanced utilization between visual and textual information. based on these findings, previous works proposed different methods for countering them in vl benchmarks. for instance, @xcite @xcite diversifies and shifts vqa's answer distribution @xcite to balance the dataset; @xcite @xcite augments images or creates counterfactual images to train more robust models on vqa; @xcite @xcite @xcite regularizes models' training with prior knowledge to avoid learning biases; @xcite directly aligns pronouns to demonstrate biases in vcr @xcite , etc. however, none of them helps us evaluate vl models' understanding on each modality. without understanding how much models understand the image, the text, or the background knowledge, it is difficult to further regularize models in training. recently, large pretrained vl models, which are mostly trained as implicit black boxes, have been dominating vl benchmarks. it is difficult to know if they understand the image and the textual information other than simply memorizing it. question-answering is the most general format for evaluating a wide range of models while having minimal requirements. @xcite @xcite annotated addi-tional questions on top of vqa questions to measure vl models' consistency on prediction. however, these works only focus on semantically lowlevel dataset like vqa and do not apply to highly semantic dataset like vcr. moreover, their data fully relies on manual annotation and thus is hard to scale. furthermore, they also fail to evaluate models' understanding across modalities. to solve these problems, we create a vl evaluation method that generates data with minimal human efforts, differentiates evaluation between modalities, and applies to highly semantic vl benchmarks. foot_0 ## multimodal-eval (me) given an input image-question(-answer) pair, me first analyzes information from the pair. then it would generate three follow-up fine-grained questions called sub-questions corresponding to three modalities: vision, text, and background knowledge. following vcr's format, each sub-question also has four answer choices with one correct answer. the vl models are expected to first answer the vcr question and then answer the three related sub-questions. through evaluating predictions of the sub-questions, we can test models' understanding across modalities. overall, me has two parts: (1) multimodal qa generator that generates related sub-questions of three modalities and (2) evaluation, which test vl models' capabilities with the sub-questions. we structure the presentation as below: the method section explains the qa generation process and the experiment section discusses the evaluation process. ## multimodal qa generator we introduce multimodal qa generator through the following steps in order: (1) retrieving related sentence statements of three modalities against the input image-text, (2) parsing the statements into three unimodal graphs and then merging them into a multimodal graph, (3) converting triplets in a multimodal graph into question and answer, (4) distractor generation, (5) adversarial filtering. ## evaluation in the following, we conduct experiments based on the proposed dataset to demonstrate (1) the existing models that perform well on vl dataset often cannot answer detailed vision, text, knowledge subquestion correctly; evaluation metrics when calculating the original q2a accuracy of vcr @xcite on n total samples, let c q2a j be an indicator variable for sample j, @xmath0, . . . , n]. if the prediction, p q2a j , is the same as the label, l q2a j , the prediction is correct and c q2a @xmath1; otherwise similarly, in our new metrics, we have indicator variables c q2s-x j for the correctness of prediction on sub-questions related to modality x, which can be vision, text, or background knowledge; similarly, we use c q2as-x j to indicate the event that both the vcr question and the sub-question corresponding to modality x are correct. lastly, c q2s j indicates the event that all the sub-questions related to sample j are predicted correctly. ## multimodal coaching for model improvement besides utilizing me data for evaluating existing vl models' performance of fine-grained understanding across modalities and prediction consistency, we also find that me can further assist existing vl models' training. ## limitation me requires the given image to have paired captions so they can be easily converted into visual statements. when absent, we can inference from a pretrained caption generator at the expense of accuracy. however, sometimes the visual caption generator may not fully captures the most salient activities in the image and thus produces trivial captions with limited contents. therefore, it would be difficult for me to extract related information from the caption to further create the visual subquestion. also, technically, for any vl dataset with imagequestion-answer pairs, me should be able to generate sub-questions from three modalities. however, if the input question is very simple and focuses on semantically low-level information. it woud be challenge for me to further extract and create sub-questions from three modalities. this study is solely based on english data and leverages linguistic structures in english so it cannot generalize to other languages.