C hart QAP ro: A More Diverse and Challenging Benchmark for Chart Question Answering (2025)

Abstract: Charts are ubiquitous, as people often use them to analyze data, answer questions, and discover critical insights. However, performing complex analytical tasks with charts requires significant perceptual and cognitive effort. Chart Question Answering (CQA) systems automate this process by enabling models to interpret and reason with visual representations of data. However, existing benchmarks like ChartQA lack real-world diversity and have recently shown performance saturation with modern large vision-language models (LVLMs). To address these limitations, we introduce ChartQAPro, a new benchmark that includes 1,341 charts from 99 diverse sources, spanning various chart types—including infographics and dashboards—and featuring 1,948 questions in various types, such as multiple-choice, conversational, hypothetical, and unanswerable questions, to better reflect real-world challenges. Our evaluations with 21 models show a substantial performance drop for LVLMs on ChartQAPro; e.g., Claude Sonnet 3.5 scores 90.5% on ChartQA but only 55.81% on ChartQAPro, underscoring the complexity of chart reasoning. We complement our findings with detailed error analyses and ablation studies, identifying key challenges and opportunities for advancing LVLMs in chart understanding and reasoning. We release ChartQAPro at https://github.com/vis-nlp/ChartQAPro.

## introduction data visualizations such as bar and line charts are very popular for analyzing data and making in-figure 1 : performance gap between chartqa @xcite and chartqapro for various lvlms. due to its real-world relevance, cqa has become a key task for evaluating recent lvlms @xcite @xcite . these lvlms have obtained remarkable performance on multimodal tasks, including cqa. for instance, on chartqa @xcite , claude sonnet 3.5 @xcite achieves an accuracy of 90.5%, while @xcite and gemini @xcite reach 85.7% and 87.2%, respectively (figure 1 ). open-figure 2 : chartqapro covers a more diverse range of questions compared to existing chart question answering datasets ( a closer look at chartqa reveals key limitations. first, its chart images lack visual diversity, coming from a few online sources like @xcite it primarily includes only bar, line, and pie charts with numeric labels directly on visual elements, reducing the need for actual visual reasoning. second, the benchmark focuses largely on factoid questions that require simple data extraction or basic arithmetic. earlier datasets @xcite @xcite suffer from similar issues, and are also curated from synthetic data or templated questions. although a recent work, charxiv @xcite , addresses some of these limitations, it relies on charts sourced exclusively from papers on arxiv, limiting visual and topical diversity, and also lacking numerous real-world question types. in contrast, real-world charts encompass diverse domains like economy, health, etc., and a wide variety of question types, including hypothetical (e.g., future price prediction), multiple-choice (e.g., in educational exams), conversational (e.g., in decisionmaking meetings) and unanswerable (e.g. due to missing data). additionally, multi-chart layouts and dashboards are often used in finance, business intelligence, and scientific reports, requiring users to analyze multiple charts simultaneously. these types of questions and complex layouts are absent from current benchmarks, suggesting that existing evaluations do not fully capture the real-world challenges in chart understanding and create an overly optimistic perception of progress in this field. to address these limitations and rigorously evaluate lvlms' on chart understanding, we present chartqapro, a comprehensive benchmark of 1341 charts sourced from 99 diverse online platforms. chartqapro includes 1948 humanwritten, human-verified question-answer pairs covering factoid, multiple-choice, conversational, hypothetical, multi-chart, and unanswerable queries, making it representative of real-world use cases (see figure 2 ). beyond bar, line, and pie charts, chartqapro features images with complex visualizations such as multi-chart layouts, infographics, and dashboards, introducing greater visual and analytical complexity. inspired by conversational and multi-document qa in text such as coqa @xcite and hotpotqa @xcite , some questions also require multi-turn interactions or referencing accompanying paragraphs, probing a broader range of multimodal reasoning skills. our evaluations reveal a sharp performance drop for both closed-and open-source models on chartqapro (figure 1 ). for example, the sota claude sonnet 3.5's accuracy falls from 90.50% to 55.81%, demonstrating that chartqapro presents a more challenging and realistic benchmark for chart understanding, and that there is ## related work chart understanding datasets numerous tasks and benchmarks have been developed to evaluate lvlms' chart understanding abilities, such as question answering @xcite , chart summarization @xcite , fact-checking @xcite , and explanation generation @xcite . among these, chart question answering is the most commonly used for evaluation. early benchmarks like stl-cqa @xcite and leaf-qa @xcite relied on synthetically generated charts and templated questions. later benchmarks, such as chartqa @xcite , plotqa @xcite , and charxiv @xcite , used real-world charts and more complex questions requiring advanced visual reasoning. however, these benchmarks extract charts from limited sources ( vision-language models for charts advances in vision-language models have significantly improved chart understanding and reasoning. these models can be categorized into: (i) closed-source, (ii) open-source general multimodal models, and (iii) chart-specific models. closed-source models @xcite achieve the highest performance on recent chart understanding benchmarks @xcite . open-source general multimodal models @xcite @xcite @xcite @xcite currently lag behind, but are rapidly closing the gap. chart-specific models @xcite @xcite demonstrate strong performance on standard benchmarks @xcite @xcite . however, their generalization to real-world chart understanding remains uncertain due to their reliance on instruction-tuning datasets with limited task diversity. chartqapro offers a more comprehensive benchmark, ensuring that model improvements reflect real progress in chart understanding abilities of these models. ## conclusion we introduced chartqapro, a more diverse and challenging benchmark for chart question answering, designed to push the limits of current vision-language models (vlms) in real-world chart reasoning. by incorporating 1341 charts from 99 sources and a broad spectrum of question types-including factoid, multiple-choice, fact-checking, conversational, and hypothetical queries-our benchmark reveals significant performance gaps between existing models and humanlevel understanding. our extensive evaluation shows that even the strongest closed-source models experience substantial performance drops, underscoring that chart reasoning remains an unsolved challenge. through detailed error analysis and ablation studies, we identify key areas for improvement, paving the way for future advancements in multimodal reasoning. we hope chartqapro serves as a catalyst for developing more robust and capable models for real-world chart comprehension. as future work, we plan on expanding the benchmark by introducing dynamic and interactive charts and dashboards, as current benchmarks only use screenshots of the charts -which often does not happen in real-world scenarios. we also aim to curate a large-scale training dataset in reasoning formats following recent advances in llm training, hoping to develop significantly more proficient chart understanding and reasoning models.