Are Large Vision Language Models up to the Challenge of Chart Comprehension and Reasoning (2024)

Abstract: Natural language is a powerful complementary modality of communication for data visualizations, such as bar and line charts. To facilitate chart-based reasoning using natural language, various downstream tasks have been introduced recently such as chart question answering, chart summarization, and fact-checking with charts. These tasks pose a unique challenge, demanding both vision-language reasoning and a nuanced understanding of chart data tables, visual encodings, and natural language instructions. Despite the recent success of Large Language Models (LLMs) across diverse NLP tasks, their abilities and limitations in the realm of data visualization remain under-explored, possibly due to their lack of multi-modal capabilities. To bridge the gap, this paper presents one of the first comprehensive evaluations of the recently developed large vision language models (LVLMs) for chart understanding and reasoning tasks. Our evaluation includes a comprehensive assessment of both closed and open-sourced LVLMs across five major chart reasoning tasks. Furthermore, we perform a qualitative evaluation of LVLMs’ performance on a diverse range of charts, aiming to provide a thorough analysis. Our findings reveal that while LVLMs demonstrate impressive abilities in generating fluent texts covering high-level data insights, they also encounter common problems like hallucinations, factual errors, and data bias. We highlight the key strengths and limitations of LVLMs in chart comprehension tasks, offering insights for future research

## introduction natural language and visualizations are two powerful complementary modalities to communicate data insights. while visualizations can be very effective in finding patterns, trends, and outliers in data, natural language can help explain the key insights in visualizations and answer questions about data @xcite . people commonly integrate text with graphical charts as this combination helps direct attention to specific elements of the chart and offers necessary explanations that might otherwise go unnoticed @xcite . moreover, interfaces that use natural language to interact with charts have other benefits, such as, enhancing chart accessibility @xcite and supporting visualization education @xcite . given the importance of chart comprehension and reasoning, researchers have introduced various tasks for the development of automated methods to aid users in chart analysis (see fig. 1 ). these include chart question answering @xcite @xcite , natural language generation for charts @xcite , and fact-checking with charts @xcite . to build automated systems for these tasks, a prevalent approach involves pre-training of models @xcite and development of frameworks without heuristic rules @xcite on language and vision tasks @xcite . however, in recent years, there has been dramatic progress in the development and widespread adoption of llms @xcite @xcite . while in the beginning, the llms were only capable of processing textual data, the rapid progress in this field has paved the way for the development of multimodal llms (in other words, lvlms), such as gpt-4v @xcite , gemini @xcite , claude-3 @xcite , phi-3 @xcite , llava @xcite , and minigpt-4 @xcite . given the rapid rise of these lvlms, there is a pressing question: are lvlms up to the challenge of chart comprehension and reasoning? in this paper, we aim to answer this question by investigating the capabilities and limitations of lvlms in the chart reasoning and comprehension domain. specifically, we examine whether the latest state-of-the-art (sota) lvlms can effectively interpret charts as well as identify key insights solely based on the chart images. this setup is crucial in real-world scenarios where the underlying data tables of charts are often unavailable. to this end, we performed extensive qualitative and quantitative analyses of the performance of lvlms on five downstream tasks across seven benchmark datasets: chartqa @xcite , @xcite , chart summarization @xcite , and vistext @xcite ), fact-checking (chartfc @xcite , chartcheck @xcite ), and chart-to-table @xcite . specifically, this work presents the first detailed analyses of lvlms on seven chart domain benchmarks, making the following main contributions: ## related work chart-related downstream tasks: several downstream tasks associated with charts have been proposed recently. chart question answering refers to answering factoid questions regarding charts @xcite @xcite @xcite . in contrast, open-ended question answering (opencqa) require explanatory responses by interpreting chart data @xcite . the chart summarization task @xcite @xcite involves creating natural language descriptions from charts, chart-to-table focuses on converting charts into data tables @xcite , and fact-checking with charts @xcite focuses on verifying factual statements related to charts. while there are other areas like infographic comprehension @xcite and science diagram qa @xcite , this study is devoted to chart-related tasks. evaluation of lvlms: openai's introduction of gpt-4v marked a significant advancement, outperforming other lvlms proposed in @xcite @xcite , particu-larly in scenarios data-scarce scenarios. google's gemini @xcite ) and anthropic's claude-3 @xcite have recently emerged as strong competitors, and microsoft's open-source phi-3 model achieved performance comparable to closed-source lvlms in multimodal tasks @xcite . while some studies compared gemini and gpt-4v models on image recognition and understanding tasks @xcite , other works have focused on solving chartrelated tasks only using data tables @xcite . additionally, some recent studies have proposed a benchmark dataset @xcite to study image-context reasoning, introduced a new llm for improved open-ended visual question-answering @xcite , and assessed gpt-4v-type models on tasks requiring structured reasoning @xcite . however, these studies address only one chart-related task (i.e., chart question answering) with quantitative analysis, whereas our work presents the most comprehensive evaluation of lvlms in the chart reasoning and comprehension domain with five downstream tasks with a wider range of qualitative and quantitative analyses. therefore, our work clearly distinguishes itself from the abovementioned works. ## conclusion and future directions this study presents a comprehensive analysis of lvlms (gpt-4v, gemini, claude, and phi-3) in interpreting and deriving insights from chart images in real-world scenarios, where data tables may not be available. through qualitative and quantitative analyses, we evaluate these models across various tasks, including zero-shot cot prompting and program-aided reasoning, assessing their impact on chart question-answering tasks. additionally, we examine lvlms' performance in open-ended text generation from chart related tasks. these analyses highlight both the strengths and limitations of lvlms and identify key research gaps. first, enhancing the generalizability and reasoning abilities of open-source lvlms in chartrelated tasks is a priority which can be explored via instruction tuning @xcite . second, there is significant potential for lvlms to produce semantically rich texts that describe high-level trends and contextual information more effectively. third, addressing key issues such as hallucinations, factual errors, and bias requires developing new benchmarks and models for detection and mitigation. we hope that the insights gained from this study will catalyze further research and advancements in the emerging area of chart reasoning.