CARETS : A Consistency And Robustness Evaluative Test Suite for VQA (2022)

Abstract: We introduce CARETS, a systematic test suite to measure consistency and robustness of modern VQA models through a series of six fine-grained capability tests. In contrast to existing VQA test sets, CARETS features balanced question generation to create pairs of instances to test models, with each pair focusing on a specific capability such as rephrasing, logical symmetry or image obfuscation. We evaluate six modern VQA systems on CARETS and identify several actionable weaknesses in model comprehension, especially with concepts such as negation, disjunction, or hypernym invariance. Interestingly, even the most sophisticated models are sensitive to aspects such as swapping the order of terms in a conjunction or varying the number of answer choices mentioned in the question. We release CARETS to be used as an extensible tool for evaluating multi-modal model robustness.

## introduction the task of visual question answering integrates the domains of computer vision and nlp by probing models' understanding of images through natural language queries. after the introduction of the visual question answering (vqa) benchmark @xcite , subsequent work identified the presence of several superficial correlations and other weaknesses latent in the vqa question gathering process @xcite , which lead to potentially optimistic evaluations when considering accuracy alone. more recently developed benchmarks @xcite @xcite explicitly avoid these weaknesses by introducing question, answer, and image balancing, or distributional shifts. while these 1 source code, data, and additional resources may be found at https://github.com/princeton-nlp/carets efforts provide more difficult benchmarks, a thorough evaluation of model capabilities requires a deeper and more detailed approach. to this end, we introduce carets -a consistency and robustness evaluative test suite for visual question answering. inspired by recent work in nlp that generates 'unit tests' for models @xcite , carets contains systematically generated vqa tests that evaluate six different capabilities that any vqa model should handle -robustness to question rephrasings, ontological reasoning, symmetrical logic, visual perturbations, question negation, and attribute antonymy. each test point in carets consists of a pair of instances which are small but strategic variations of each other either visually or in the question's text. this allows us to conduct fine-grained capability evaluations beyond just measuring high-level accuracy scores. across tests, we generate over 190,000 instance pairs in total using a programmatic approach that fills in templates (from nearly 200 templates in total) using ground-truth scene graphs @xcite from the gqa @xcite ) validation split. we then evaluate six modern vqa models on each test using several metrics: overall accuracy, self-consistency and comprehensive accuracy. self-consistency measures models' ability to maintain their answer across question variations, while comprehensive accuracy estimates their ability to answer all instance variants correctly. our experiments reveal several interesting findings: (1) most modern vqa systems achieve only middling self-consistency (∼60-80%) which is further not always correlated with their accuracy, (2) all models struggle to comprehend the concept of negation (self-consistency of 18-28% and comprehensive @xmath0, and (3) even simple perturbations like changing the order of choices in the question text can induce a substantial drop (10-15%) in performance. moreover, even stateof-the-art models like lxmert @xcite are highly sensitive to the type of questions (binary vs multi-choice) and the number of choices provided. these results reveal several shortcomings in modern vqa systems and hint at potential areas for future improvement. going beyond our current discoveries, carets is an extensible framework and may be easily extended by adding new capability tests for fine-grained evaluation of future models. ## fine-grained capability tests our goal is to provide a testbed for fine-grained evaluation of vqa models' capabilities. to do so, we generate multiple tests, each corresponding to a specific model capability. in contrast to standard vqa benchmarks @xcite @xcite , our test sets consist of a pair of original and perturbed instances (i 1 , q 1 , a 1 ), (i 2 , q 2 , a 2 ) , each with an image, a question and an answer. the two instances within a pair differ from each other in a minimal yet carefully constructed way to hone in on a particular capability, similar to blimp @xcite . a model is then evaluated on its overall accuracy, self-consistency (ability to produce consistent, even if wrong, answers to both instances within a pair), and comprehensive accuracy (ability to answer consistently and correctly for an instance pair). overall, we create six tests corresponding to key capabilities. we provide a high-level description of each test here and describe generation details in section 4. first, we create four invariance tests 2 that use variations of the question phrasing and expect the model to produce the same answer to both questions within an instance pair: we also create directional expectation tests to measure model behavior on instance pairs where the answer is expected to change: ## dataset generation each of the six test datasets start with the generation of 'original', unperturbed instances (i 1 , q 1 , a 1 ) (section 4.1). then, for each such instance, we generate a variation (i 2 , q 2 , a 2 ) by either perturbing the original question q 1 or image i 1 (section 4.2). further, each test set is composed of a diverse set of questions. these may broadly be grouped into verification (or binary) questions, with expected answers being either yes or no, and multi-choice questions, with expected answers derived from a list of objects or attributes provided in the question. ## conclusion & future work in this work, we have developed carets -a new test suite for capability-focused robustness testing and comprehension evaluation of visual question answering (vqa) models. carets consists of six different tests that use instance pairs to evaluate models on their understanding of various linguistic and visual concepts. using this test suite, we evaluated six modern vqa systems to reveal several inconsistencies in state-of-theart models and provide a fine-grained view of their comprehension of different visuo-linguistic concepts. quite surprisingly, we find that even state-of-the-art models struggle with concepts like negation, disjunction, order invariance and multichoice questions. carets can also support the addition of more tests in the future and we view it as a platform for continually stress testing and improving vqa models. carets emulates previous work in using text templates to generate questions and their textual perturbations @xcite @xcite . the use of templates to generate perturbations is motivated by the desire to maintain the grounded integrity of generations, ensuring that they remain relevant and that the generated label is true in the context of the subject image. while we have sought to generate a diverse language set through using a large number of templates (nearly 200 in total), there are some limitations to this approach. an improvement to our approach may be able to generate more sophisticated questions and perturbations through conditional text generation @xcite @xcite while also preserving our other motivations, such as atomicity and grounded relevancy.