C hart I nsights: Evaluating Multimodal Large Language Models for Low-Level Chart Question Answering (2024)

Abstract: Chart question answering (ChartQA) tasks play a critical role in interpreting and extracting insights from visualization charts. While recent advancements in multimodal large language models (MLLMs) like GPT-4o have shown promise in high-level ChartQA tasks, such as chart captioning, their effectiveness in low-level ChartQA tasks (*e.g.*, identifying correlations) remains underexplored.In this paper, we address this gap by evaluating MLLMs on low-level ChartQA using a newly curated dataset, *ChartInsights*, which consists of 22,347 (chart, task, query, answer) covering 10 data analysis tasks across 7 chart types. We systematically evaluate 19 advanced MLLMs, including 12 open-source and 7 closed-source models. The average accuracy rate across these models is 39.8%, with GPT-4o achieving the highest accuracy at 69.17%.To further explore the limitations of MLLMs in low-level ChartQA, we conduct experiments that alter visual elements of charts (*e.g.*, changing color schemes, adding image noise) to assess their impact on the task effectiveness. Furthermore, we propose a new textual prompt strategy, *Chain-of-Charts*, tailored for low-level ChartQA tasks, which boosts performance by 14.41%, achieving an accuracy of 83.58%. Finally, incorporating a visual prompt strategy that directs attention to relevant visual elements further improves accuracy to 84.32%.

## introduction visualization charts can effectively convey data insights, but the abundance of information they provide makes it challenging for users to efficiently and accurately extract desired information @xcite @xcite . automated chart question answering (chartqa) @xcite is crucial to * equal contribution † yuyu luo is the corresponding author help users pinpoint relevant information based on their intents @xcite . high-level and low-level chartqa tasks. chartqa tasks can generally be categorized into two types: high-level tasks and low-level tasks. high-level tasks focus on questions that require understanding the overall context or summary of the chart, involving broader, goal-oriented inquiries that seek to understand overarching trends or patterns. in contrast, low-level tasks focus on specific, detail-oriented inquiries that seek precise data points or comparisons within the chart, involving straightforward, factual information retrieval @xcite . for example, as shown in figure 1 , given the same line chart showing rainfall in different months of the same year, a high-level task might ask about the cyclical or seasonal trends in the chart, while a lowlevel task would be more focused on the data itself in the chart, such as asking how much total rainfall there was in july and august. traditionally, chartqa has been a challenging problem due to the limited capabilities in natural language understanding and the high complexity of chart reasoning @xcite @xcite . fortunately, recent advancements in multimodal large language models (mllms) have made it possible for users to interact with systems using natural language to extract specific information from data across various modalities. this progress has illuminated new possibilities for chartqa on different levels of tasks. high-level chartqa dvqa @xcite ) 3 1 300k 3.5m 11.7 chartqa @xcite 4 3 4.8k 9.6k 2 mix figureqa @xcite ) 6 5 120k 1.5m 12.9 chartllama @xcite 7 10 11k 160k 14.5 chartbench @xcite ) 4 11 2.1k 16.8k 8 low-level chartqa chartinsights (ours) 10 7 2k 22k 11.2 prior art: high-level chartqa with mllms. recent studies have explored the capabilities of mllms in performing high-level chartqa @xcite @xcite @xcite @xcite . the findings reveal that stateof-the-art mllms like gpt-4o have demonstrated promising results in addressing high-level tasks, and have outlined future research directions. our focus: low-level chartqa with mllms. existing evaluations of chartqa primarily focus on high-level tasks, such as chart captioning, while overlooking low-level chartqa tasks (e.g., characterizing distributions) that humans frequently encounter in daily life. specifically, studies from the visualization and visual analysis community have well-defined 10 widely used low-level chartqa tasks @xcite . thus, our study seeks to systematically evaluate the effectiveness of mllms in addressing these 10 low-level chartqa tasks. ## chartinsights dataset since no dataset exists for low-level chartqa tasks, we construct a large-scale dataset, chartinsights, to systematically evaluate the performance of mllms in these tasks. the construction details of chartinsights are provided in appendix a.2. ## conclusion in this paper, we curate a large-scale dataset, chartinsights, specifically designed for low-level chartqa tasks. to evaluate the capabilities of mllms on these tasks, we conduct a series of experiments using 19 widely used mllms from multiple perspectives. specifically, we investigate the impact of chart variants and visual prompts on performance, demonstrating the importance of chart quality and visual attention. we also propose a new textual prompt strategy, named chainof-charts, to harness the capabilities of mllms for low-level chartqa. by incorporating visual prompts, we achieve the best average accuracy of 84.32% using gpt-4o. future work can explore incorporating data prompts and multi-agent frameworks to further enhance the effectiveness of mllms in diverse chartqa tasks.