Do LVLM s Understand Charts? Analyzing and Correcting Factual Errors in Chart Captioning (2024)

Abstract: Advances in large vision-language models (LVLMs) have led to significant progress in generating natural language descriptions for visual contents. These powerful models are known for producing texts that are factually inconsistent with the visual input. While some efforts mitigate such inconsistencies in natural image captioning, the factuality of generated captions for structured visuals, such as charts, has not received as much scrutiny. This work introduces a comprehensive typology of factual errors in generated chart captions. A large-scale human annotation effort provides insight into the error patterns in captions generated by various models, ultimately forming the foundation of a dataset, CHOCOLATE. Our analysis reveals that even advanced models like GPT-4V frequently produce captions laced with factual inaccuracies. To combat this, we establish the task of Chart Caption Factual Error Correction and introduce CHARTVE, a visual entailment model that outperforms current LVLMs in evaluating caption factuality. Furthermore, we propose C2TFEC, an interpretable two-stage framework that excels at correcting factual errors. This work inaugurates a new domain in factual error correction for chart captions, presenting a novel evaluation metric, and demonstrating an effective approach to ensuring the factuality of generated chart captions. The code and data as well as the continuously updated benchmark can be found at: https://khuangaf.github.io/CHOCOLATE/.

## introduction large vision-language models (lvlms) have recently shown impressive capabilities in generating natural language descriptions of visual content like images, videos and charts @xcite @xcite . chart captioning is particularly important for data analysts, business analysts, and journalists who rely on accurate chart interpretations for decision-making and reporting. however, no prior work has studied the factuality foot_0 of the generated captions. given that factuality is vital for credibility in applications of chart captioning in news articles @xcite , educational resources @xcite , and social media @xcite , examining the truthfulness of generated captions is a critical concern. to understand the factual errors in chart captioning models, we introduce a typology of factual errors for the chart domain. using this scheme, we conduct a large-scale human annotation study to analyze the distributions of various error types, such as value error and label error, in captions from various models, from task-specific fine-tuned models to lvlms (see the prevalence of factual inconsistencies observed in the generated captions by various models underscores the urgent need to mitigate the factual errors of such models. hence, we introduce a new task, chart caption factual error correction ( §3), which presents a novel challenge of rectifying now that we have set up the task, we turn to the challenge of how to effectively correct factual errors in chart captions? we propose c2tfec ( §4), an interpretable two-step framework that decomposes visual reasoning into image-to-structure rendering and text-based reasoning. c2tfec first transforms the input chart into a structured data table representation. grounded in this extracted tabular data, the second component then identifies and fixes any factual inconsistencies in the generated caption through an interpretable reasoning process. our experiments demonstrate that this explicit decomposition enables more reliable factuality corrections compared to end-to-end approaches. the intermediate symbolic representation acts as an effective bridge between charts and captions, enabling c2tfec to significantly outperform competitive baselines including gpt-4v ( §6). in summary, our contributions are as follows: ## analyzing factual errors to understand the capabilities of existing models in summarizing key information from charts, we conduct a large-scale analysis on six most advanced chart captioning models on the vistext @xcite and @xcite datasets. to facilitate this process, we introduce an error typology, as illustrated in §2.1. upon gathering human annotations, we present a detailed analysis of different captioning models ( §2.2) and discuss the quality of the collected data ( §2.3). ## the chart caption factual error correction task the dataset collected in §2 enables us to study the chart caption factual error correction task. in this section, we first formally provide the definition of this task ( §3.1) and propose an effective reference-free evaluation metric based on chart vi-sual entailment ( §3.2). ## methodology in correcting factual errors in generated captions, we propose c2tfec, a two-step, interpretable framework, as shown in figure 2 . c2tfec first transforms input charts into data tables ( §4.1), then rectifies errors in the caption using the tabular data (4.2). this framework is motivated by our analysis on "deplot + gpt-4", which shows that a notable proportion of errors in caption generation originated from the deplot component. to mitigate this, we develop a stronger chart-to-table model based on unichart, significantly improved with expansive fine-tuning datasets. the advantage of c2tfec is its ability to harness the reasoning strengths of gpt-4 to faithfully correct errors, boosting caption factuality. the training data for our chart-to-table model is sourced from datasets including vistext, chart-to-text, chartqa, and plotqa, where we repurpose original charts and underlying data tables for our model's training. we collected a total of 65k instances with a train:dev:test split of 61:2:2. similar to deplot @xcite , our model is also trained to generate chart titles, enhancing its ability to contextualize the data represented in table form. let m denote our proposed model. for a given chart figure e, the model autoregressively generates a chart title t and a corresponding table a (i.e. t , @xmath0. ## experimental settings to assess c2tfecs ability in factual error correction for chart captions, we experiment on the chocolate dataset. datasets our chocolate dataset includes 1,187 chart-caption pairs with factually consistent and inconsistent captions, as detailed in §2. it is split into lvlm, llm, and ft, reflecting the diversity of models that generated the captions. baselines since chocolate does not comprise training data, we compare c2tfec against zeroshot capable lvlms and llms, including lvlms, llava-1.5-13b, gpt-4v, bard, as well as deplot + gpt-4. for a fairer comparison between our approach and deplot, we continue fine-tuning deplot for an additional 5,000 steps on vistext, an approach which has been shown effective for adapting models to unseen domains @xcite . we denote this model as deplot cft . the prompts used for each model are described in appendix e. ## conclusion our study exposes the prevalent issue of factual errors in chart captions generated by various chart captioning models and introduces chocolate to scrutinize these errors. we establish the chart caption factual error correction task to propel the creation of trustworthy captioning systems and present chartve, an evaluation model surpassing lvlms in mirroring human assessments of caption factuality. our two-stage correction framework, c2tfec, provides an interpretable means of improving caption factuality by transforming visual data into structured tables for more faithful error corrections. our work marks an essential step in ensuring verifiable and trustworthy chart captions. future directions include extending our approach to multimodal contexts beyond charts, developing more sophisticated error detection and correction algorithms, and creating datasets covering a broader range of visual content.