C hart C heck: Explainable Fact-Checking over Real-World Chart Images (2024)

Abstract: Whilst fact verification has attracted substantial interest in the natural language processing community, verifying misinforming statements against data visualizations such as charts has so far been overlooked. Charts are commonly used in the real-world to summarize and com municate key information, but they can also be easily misused to spread misinformation and promote certain agendas. In this paper, we introduce ChartCheck, a novel, large-scale dataset for explainable fact-checking against real-world charts, consisting of 1.7k charts and 10.5k human-written claims and explanations. We systematically evaluate ChartCheck using vision-language and chart-to-table models, and propose a baseline to the community. Finally, we study chart reasoning types and visual attributes that pose a challenge to these models.

## introduction data visualizations (e.g. bar charts, pie charts, line graphs) are common in real-world data sources and are frequently used in scientific documents, textbooks, news articles, and social media to summarize key information in a visual form, information which is often not fully repeated in the associated text @xcite . however, data visualizations are also commonly misused to spread misinformation, 2 being included in political and business advertisements, or spread on social media, with the aim to convince the audience towards certain agendas @xcite . for example, during the covid pandemic, the term "counter-visualizations" was coined for charts circulating on social media platforms to counter covid-related health measures @xcite (see fig. 2 ). charts were also (mis-)used by brexit campaigners during the referendum. 3 1 code and data available at https://github.com/ mubasharaak/chartcheck 2 see fig. 2 and appendix for examples. 3 see fig. 10 and 11 in the appendix. claim: the percentage of votes for loprez obrador decreased over time as more poll stations were counted. explanation: the chart shows a generally downward trend in the percentage of votes for obrador as more poll stations were counted. as more votes were counted, the other candidate gained a larger percentage of the overall vote. figure 1 : a dataset instance labelled as supported. example includes the claim, the chart, its caption, and an explanation justifying why the claim is supported. while previous research has focused on misleading chart design (i.e. truncated axis) @xcite , a less studied but prevalent issue is misleading statements that exploit information fallacies during chart interpretation. verifying these statements requires extracting information from charts that use tightly integrated text and visual elements to represent information. these visual elements consist of various lines and shapes, and have different colors, scales, angles, and orientations. moreover, various reasoning skills, including numeracy and language understanding, are required to successfully verify statements against charts. in this paper, we introduce chartcheck, a novel dataset for explainable fact-checking against data visualizations. chartcheck is the first large-scale dataset with 1.7k real-world charts extracted from the web and 10.5k human-written claims and explanations (see fig. 1 for an example). different to previous datasets, chartcheck provides explanations as justifications for claim verification. explaining fact-checking decisions is an important aspect as a major goal of fact-checking is to convince readers, and debunking misinformation by only labelling it as false is often not sufficient @xcite . to collect the data, we apply a four-step crowdsourcing pipeline to filter out noisy images, write claims and explanations, evaluate the data, and apply a final verification check through expert annotators. we evaluate state-of-the-art (sota) models in a finetuning, few-and zero-shot setting. hereby, we consider two baseline architectures: (1) visionlanguage models (vlms) and (2) chart-to-table architectures with separate models for each subproblem: chart translation (chart-to-table), claim verification, and explanation generation. our bestperforming chart-to-table baseline reaches 73.8 accuracy and lags far behind human performance. we annotate a subset of chartcheck with (i) reasoning types humans apply for chart understanding @xcite and (ii) visual attributes common in charts. we evaluate which reasoning types and chart attributes pose a challenge for sota models, as well as common failure types related to explanation generation. we evaluate modelgenerate explanations for factuality, semantic coverage, coherence, readability, and redundancy. overall, our results show that models generate very fluent, readable and coherent text, but show major limitations w.r.t. understanding and reasoning over charts to generate correct explanations. hence, chartcheck is a challenging problem and will stimulate progress on fact-checking against charts. to summarise, our contributions are as follows: 1) we propose chartcheck, the first chart dataset for explainable fact-verification with 1.7k realworld charts and 10.5k human-written claims and explanations. ## the need for chart fact-checking charts in the real-world. data visualizations are commonly consumed on a daily basis, e.g. in news articles, text books, scientific papers, and on the in-figure 2 : twitter post related to covid misinformation claiming that number of deaths in sweden did not increase through the pandemic although data for the year 2020 is not complete @xcite . ## chartcheck: dataset creation fig. 3 provides an overview of the dataset creation pipeline. starting with chart extraction from the web (step 1), we applied three crowdsourcing tasks (step 2 -4), before post-processing and validating the collected data (step 5). ## experiments and results we define the explainable fact-verification task against chart evidence as follows. each dataset entry (s i , v i , c i , y i , e i ) comprises a natural language statement s i , a chart image v i , a caption c i accompanying the chart image, a verdict label y @xmath0, 1}, and an explanation e i in natural language. the statement s i is a claim that is either supported (y @xmath1. 1 ). ## conclusion we introduce chartcheck, the first dataset for explainable fact-checking with 1.7k real-world chart and 10.5k human written claims and explanations. we evaluate sota models, including chart-to-table baselines and vlms, in a few-/zero-shot and finetuned setting. our best baseline yields 73.8 accuracy, lagging far behind human performance. moreover, we study reasoning types and visual attributes that pose a challenge to sota models, as well as failures related to model-generated explanations.