Beyond Accuracy: A Consolidated Tool for Visual Question Answering Benchmarking (2021)

Abstract: On the way towards general Visual Question Answering (VQA) systems that are able to answer arbitrary questions, the need arises for evaluation beyond single-metric leaderboards for specific datasets. To this end, we propose a browser-based benchmarking tool for researchers and challenge organizers, with an API for easy integration of new models and datasets to keep up with the fast-changing landscape of VQA. Our tool helps test generalization capabilities of models across multiple datasets, evaluating not just accuracy, but also performance in more realistic real-world scenarios such as robustness to input noise. Additionally, we include metrics that measure biases and uncertainty, to further explain model behavior. Interactive filtering facilitates discovery of problematic behavior, down to the data sample level. As proof of concept, we perform a case study on four models. We find that state-of-the-art VQA models are optimized for specific tasks or datasets, but fail to generalize even to other in-domain test sets, for example they can not recognize text in images. Our metrics allow us to quantify which image and question embeddings provide most robustness to a model. All code s publicly available.

## introduction vqa refers to the multi-modal task of answering free-form, natural language questions about images -a task sometimes referred to as a visual turing test @xcite . the number and variety of datasets for evaluating such systems has continued to increase over the last years @xcite @xcite @xcite . these datasets aim to test models' abilities with respect to different skills, such as commonsense or external knowledge reasoning, visual reasoning, or reading text in images. traditionally, evaluation relies solely on answering accuracy. however, it is misleading to believe that a single number, like high accuracy on a given benchmark, corresponds to a system's ability to answer arbitrary questions with high quality. each dataset contains biases which state-of-the-art deep neural networks are prone to exploit, resulting in higher accuracy scores @xcite @xcite . thus, most vqa models, if evaluated on multiple benchmarks at all, are re-trained per dataset to achieve higher numbers in specialized leaderboards. further shortcomings of current leaderboards include ignoring prediction cost and robustness, as discussed in @xcite ) for nlp. in vqa, we need even more specialized evaluation due to the challenges inherent to open-ended, multi-modal reasoning. in order to successfully develop vqa systems that are able to answer arbitrary questions with human-like performance, we should overcome the previously mentioned shortcomings of current leaderboards as one of the first essential steps. to this end, we propose a benchmarking tool, to our knowledge the first of its kind in the vqa domain, that goes beyond current leaderboard evaluations. it follows the four following principles: ## vqa benchmark tool our tool facilitates global evaluation of model performance w.r.t general and specific tasks (generalizability), such as real-world images and reading capabilities. to simplify integration of future benchmark datasets and models, we provide a well-documented python api. we measure modelinherent properties, such as biases and uncertainty (explainability) as well as robustness against input perturbations (realism). model behavior can be further inspected using interactive diagrams and filtering methods for all metrics, supporting sample-level exploration of suspicious model behavior (interactivity). all data is collected posthoc and can be explored in a web application, eliminating the need to re-train existing models. ## case study as a case study, we explore a range of models from well-established, previously high ranking entries in the vqa2 competition to more recent, transformer-based architectures and report the insights we gained by inspecting them with our tool. ## conclusion our proposed benchmarking tool is the first of its kind in the domain of vqa and addresses the problems of current single-metric leaderboards in this domain. it provides easy to use and fast comparison of integrated models on a global level. the performance of each model is evaluated across multiple special-purpose as well as general-purpose datasets to test generalizability and capabilities. each model can be quantified by metrics such as accuracy, biases, robustness, and uncertainty, revealing strengths and weaknesses w.r.t to given tasks, i.e. measuring the properties models offer as well as their real-world robustness. exploration via filtering can be used to identify suspicious behaviour down to single data sample level. through this, our tool provides deeper insights into the strengths and weaknesses of each model across tasks and metrics and how architectural choices can affect behavior, encouraging researchers to develop vqa systems with rich sets of abilities that stand up to real-world in the future, we plan to extend this tool with new datasets as they are released. moreover, we are looking for more metrics for model evaluation as well as more detailed dataset analysis, e.g. answer space overlap. last but not least, interactivity could be extended towards live model feedback, allowing to change inputs, e.g. the image noise level, and observe model outputs at runtime.