M ulti C hart QA : Benchmarking Vision-Language Models on Multi-Chart Problems (2025)

Abstract: Multimodal Large Language Models (MLLMs) have demonstrated impressive abilities across various tasks, including visual question answering and chart comprehension, yet existing benchmarks for chart-related tasks fall short in capturing the complexity of real-world multi-chart scenarios. Current benchmarks primarily focus on single-chart tasks, neglecting the multi-hop reasoning required to extract and integrate information from multiple charts, which is essential in practical applications. To fill this gap, we introduce MultiChartQA, a benchmark that evaluates MLLMs’ capabilities in four key areas: direct question answering, parallel question answering, comparative reasoning, and sequential reasoning. Our evaluation of a wide range of MLLMs reveals significant performance gaps compared to humans. These results highlight the challenges in multi-chart comprehension and the potential of MultiChartQA to drive advancements in this field. Our code and data are available at https://github.com/Zivenzhu/Multi-chart-QA.

## introduction multimodal large language models (mllms) have received significant attention due to their impressive capabilities across a broad set of visionlanguage tasks, such as image captioning @xcite , multimodal dialogue systems @xcite , and visual question answering (vqa) @xcite @xcite . among their varied applications, the comprehension of visual chart data, such as bar and line charts, holds particular significance @xcite . such data is prevalent in real-world scenarios, including academic papers and analytical reports, which makes understanding and reasoning over these charts an essential skill * equal contribution. for mllms. as a result, several efforts have been made in evaluating and improving model performance on chart-related tasks @xcite @xcite . despite these efforts, existing chart understanding benchmarks are limited to single-chart scenarios @xcite @xcite @xcite . this narrow focus fails to cover evaluation settings involving multiple charts, which are common in real-world applications such as news articles and scientific documents. such cases usually require mllms to integrate information from different charts and perform multi-hop reasoning to derive the answer to the question. for instance, figure 1 demonstrates an example of a multi-chart question from a research paper. to determine whether the "exp avg gap" for the optimizer "adamw" exceeds that of "sgdm" under the gpt2 setting, information from multiple charts and sub-charts must be extracted and combined through a three-hop reasoning process to arrive at the correct answer. such complex scenarios are hardly covered by single-chart benchmarks. in fact, solving these cases requires advanced capabilities of mllms, such as multi-image encoding, precise information localization, and multi-hop reasoning. the absence of complex multi-chart evaluation data hinders the further development of these advanced mllm capabilities. to bridge this gap, we introduce multi-chartqa, a newly collected benchmark designed to evaluate the capabilities of mllms on multichart scenarios. our benchmark draws from a diverse set of public resources containing multi-chart articles, including arxiv, oecd, our world in data, pew research center, usafacts, the world economic forum, data commons, international energy agency, gallup, and gapminder. using the charts collected from these sources, we design four different question types, each paired with multiple charts from the same article as input: together, these four tasks provide a comprehensive evaluation of mllms' multi-chart processing abilities. all questions and answers in the benchmark are manually annotated to ensure their high quality. we evaluate the performance of 20 mainstream mllms on multichartqa. the results reveal a significant limitation in mllms' ability to process multiple charts, with performance trailing behind humans by a large margin. furthermore, there is a notable gap between the performance of proprietary models and their open-source counterparts. further analysis conducted on our benchmark highlights the benefits of using chain-of-thought reasoning, as well as the models' reliance on chart references for information retrieval. an in-depth error analysis reveals that mllms still struggle with visual perception and multi-step reasoning, especially when color or spatial constraints are involved. we believe that multichartqa will illuminate the evaluation of multi-chart capabilities in mllms and foster advancements in this field in the near future. ## multichartqa multichartqa is an extensive and demanding benchmark that features real-world charts and complex reasoning questions. to cover charts with various topics and a diverse range of styles, we curate chart sets from a variety of real-world sources ( §3.1). based on four core competencies in chart comprehension, we design four categories of question-answer pairs ( §3.2). we detail the evaluation metrics in §3.3. ultimately, the multichartqa benchmark comprises 1,370 charts and 2,000 questions. ## conclusion we introduce multichartqa, a comprehensive benchmark for evaluating multi-chart understanding using semantically related chart sets and questions spanning multiple categories. testing 20 mllms reveals a significant performance gap between closed-source and open-source models. incorporating chain-of-thought reasoning notably improves accuracy. multichartqa highlights crucial avenues for advancing mllms in multi-chart reasoning tasks.