Plot Twist: Multimodal Models Don’t Comprehend Simple Chart Details (2024)

Abstract: Recent advances in multimodal models show remarkable performance in real-world benchmarks for chart and figure understanding like ChartQA that involve interpreting trends, comparing data points, and extracting insights from visuals.In this paper, we investigate the extent to which these models truly comprehend the underlying information in charts by posing direct, elementary questions about simple features such as axes ranges and values to examine their fundamental visual understanding abilities in the context of charts.Our questions are applied to two sets of figures: synthetic and real-world.The empirical evaluation of 5 popular multimodal models on our dataset reveals shortfalls in understanding charts and figures, contrary to what their performance on complex benchmarks might suggest.For instance, Gemini Pro Vision only achieves 57.9% accuracy on our elementary set of questions on real-world plots, while other popular multimodal models showed similar or less performance.This work highlights an important limitation of current multimodal models, and cautions against overly optimistic interpretations of their abilities based on results of canonical evaluations.

## introduction assessing chart understanding capabilities offers a crucial benchmark for evaluating foundational models' reasoning skills beyond text. significant efforts have been made to develop benchmarks for chart understanding, such as chartqa, that features complex, human-written questions reflecting real-world applications @xcite . multimodal models have recently made significant progress on these evaluation benchmarks @xcite @xcite . while these models perform well on complex tasks, how do they fare with more elementary aspects of chart understanding? can they reliably answer basic questions about the chart? previous work has showed that real world datasets -while very useful for ensuring the practical application -can often contain statistical patterns such that model can do well without fully understanding the relevant capability @xcite . moreover, testing basic capabilities can highlight important limitations of models that do not appear in complex benchmarks @xcite . complex capabilities examined in these real-world benchmarks, such as obtaining insights from visualizations, are also made up of many steps: understanding the image, domain knowledge, and reasoning. this makes it harder to diagnose the cause for failures. in this work, we probe multimodal models to understand whether they can answer elementary questions about the specific visual content in charts. this is a core capability that is essential for any model claiming proficiency in chart comprehension. we evaluate this understanding by constructing elementary probing questions. these elementary questions include straightforward questions that measure fundamental skills like identifying axis extremes and extracting plot values on synthetic plots. we first pose these elementary questions on basic, synthetic plots. then, we select a subset of real-world chartqa test plots and pose our simple questions to them. this allows us to directly compare model performances on complex chartqa queries versus performance on our elementary ones (examples in figure 1 ). our findings uncover shortcomings in these models regarding fundamental aspects of chart understanding. for example, pali-3 only gets 37.7% accuracy on our straightforward questions on the real-world plots. moreover, other models such as gemini pro vision and gpt-4v also get less than 60% performance. we further evaluate the robustness of these models and find that more powerful models, such as gemini pro vision and gpt-4v, are often susceptible to the presence of text annotations over the actual data presented in the plots, which negatively affects their accuracy. this study highlights critical limitations of current multimodal models and underscores the importance of rigorous and thorough testing especially given limited public knowledge of the data used to train them. 1 ## setup in the following section, we introduce our evaluation method, designed specifically to assess the understanding of elementary features in both synthetic and real-world chart and figure understanding by multimodal models. ## results models struggle to answer basic synthetic chart questions reliably. we initially assess model performance on our synthetic subset. as demonstrated in ## robustness tests one key aspect of chart understanding is the models' resilience to visual changes that do not affect the informational content but only the visual presentation, such as the choice of the plotting library or variations in phrasing. our synthetic subset allows us to comprehensively evaluate model robustness against these changes. we make targeted visual modifications to charts to assess the models' ability to maintain accurate interpretation despite superficial alterations. this evaluation is crucial for determining the real-world utility and robustness of multimodal models in chart comprehension. ## conclusion in this paper, we present a diagnostic method for evaluating multimodal foundation models, with a focus on chart understanding capabilities. our approach combines the precision of controlled synthetic evaluations with the real-world relevance of natural data scenarios. this dual strategy is essential in the current landscape, where models often lack transparency regarding training data and operate behind apis. our evaluation method complements real-world datasets like chartqa, emphasizing the need to look beyond unified metrics and thoroughly assess models to identify their failure modes. by exposing subtle limitations, our method lays the groundwork for more effective benchmarks that accurately capture previously hidden model weaknesses. we believe this work will inspire further innovation in the field, promoting a holistic and nuanced approach to model evaluation.