Unraveling the Truth: Do VLM s really Understand Charts? A Deep Dive into Consistency and Robustness (2024)

Abstract: Chart question answering (CQA) is a crucial area of Visual Language Understanding. However, the robustness and consistency of current Visual Language Models (VLMs) in this field remain under-explored. This paper evaluates state-of-the-art VLMs on comprehensive datasets, developed specifically for this study, encompassing diverse question categories and chart formats. We investigate two key aspects: 1) the models’ ability to handle varying levels of chart and question complexity, and 2) their robustness across different visual representations of the same underlying data. Our analysis reveals significant performance variations based on question and chart types, highlighting both strengths and weaknesses of current models. Additionally, we identify areas for improvement and propose future research directions to build more robust and reliable CQA systems. This study sheds light on the limitations of current models and paves the way for future advancements in the field.

## introduction chart question answering (cqa) @xcite has emerged as a critical area within the field of visual language understanding (vlu) @xcite , aiming to equip machines with the ability to comprehend and answer questions based on data visualizations. while recent advancements in vision language models (vlms) and multimodal large language models (mllms) have yielded impressive performance improvements in cqa @xcite @xcite @xcite @xcite , their true capabilities remain obscure in uncertainty. this paper delves into an insightful analysis of the robustness and consistency * contributed equally, ‡ corresponding author (work done while at upenn) ## initial dataset this section highlights the dataset preparation process employed to analyze the performance of cqa models across a spectrum of chart types and question complexities. ## can models reasons consistently? this section presents our findings and analysis on the performance of various chart question answering (cqa) models across different chart types and question complexities. ## are models robust on cqa? another crucial aspect of our analysis involves investigating the robustness and consistency of these models across different visual representations of the same underlying data. through the help of this probing, we aim to understand if model performance remains stable when presented with variations in chart types, styles, or aesthetics while conveying the same information. figure 3 illustrates how an original chart can be converted into stair plots, bar charts, stacked representations, and more. these variations may differ in color schemes, patterns, legend positioning, and other chart-specific details like legend orientation and grid sizes on the x-axis and y-axis. examining these variations can offer deeper insights into the data and improve the clarity of the visualizations. ## related work chart comprehension and question answering (cqa) are critical domains with a growing body of research. while existing cqa datasets assess models' advanced reasoning capabilities over charts, many face significant limitations. these include small dataset sizes @xcite , reliance on template-based questions and synthetically generated charts @xcite @xcite , restriction to specific domains @xcite @xcite , or focusing solely on open-domain question answering @xcite . even the current state-of-the-art dataset, chart qa @xcite , has limitations due to the lack of classification labels for more meaningful analysis, and limited variation in chart types. more recent datasets, such as chartx @xcite , have expanded the range of chart types analyzed. chartbench @xcite and mmc @xcite focus on large-scale datasets with more diverse chart types. a very recent work, charxiv @xcite , provides extensive evaluations across a range of charts and questions, including both reasoningbased and descriptive queries. they also perform ablation studies by modifying charts and questions. however, to the best of our knowledge, ro-bustcqa is the first dataset to systematically perturb all elements within a chart, enabling finegrained analysis of factors affecting model performance. additionally, we conduct a detailed analysis of the chart qa dataset based on question and chart complexity, which has not been done at this level of detail before. modeling approaches for charts various approaches have been developed for chart modeling. this includes models specifically designed for chart comprehension and reasoning, built with the end-toend goal of reasoning over charts @xcite @xcite , as well as models that convert charts into intermediate table formats @xcite , enabling reasoning by generalized large language models (llms) through chain of thought @xcite or program of thought @xcite prompting. additionally, there are generalized models used for multi-modal reasoning tasks, including chart comprehension @xcite @xcite @xcite . recent efforts have also focused on developing smaller, yet accurate models for this task @xcite . while these approaches have shown significant progress, their specific failure points remain unclear. a recent study @xcite analyzed the performance of gpt-4v and gemini, providing a broad evaluation of these models across various chart comprehension tasks, including question answering, summarization, and fact-checking. in contrast, our work focuses specifically on cqa across a broader range of models, offering an in-depth analysis of the question and chart types contributing to model failures. our contribution identifies the exact question types and chart elements that lead to model errors, offering key insights to improve model performance. ## conclusion this research introduces chartqa-split and ro-bustcqa, the first datasets dedicated to understanding model consistency across complexities and robustness to visual perturbations in chart question answering. our evaluation of sota models, including baselines and vlms/mllms, using a zeroshot chain-of-thought setting, reveals significant challenges in both areas. we perform an in-depth analysis of model weaknesses and identify key areas for improvement, such as enhancing data extraction for non-annotated charts and developing models that can effectively interpret complex visual information, taking every possible visual cue into consideration. our work provides a foundation for future research in developing more robust and reliable chart question answering systems. future directions. our perturbation analysis provides a nuanced understanding of model performance by revealing both universal and modelspecific vulnerabilities and strengths. this insight drives targeted improvements: model pretraining: focusing on perturbations that affect models allows for effective fine-tuning to address weaknesses. perturbation-aware training: integrating specific perturbations during training enhances overall robustness, helping models develop resilience against challenges. interpretable models: understanding the impact of perturbations aids in debugging and building explainable models, fostering the development of reliable and transparent chart understanding and reasoning systems.