Reading and Reasoning over Chart Images for Evidence-based Automated Fact-Checking (2023)

Abstract: Evidence data for automated fact-checking (AFC) can be in multiple modalities such as text, tables, images, audio, or video. While there is increasing interest in using images for AFC, previous works mostly focus on detecting manipulated or fake images. We propose a novel task, chart-based fact-checking, and introduce ChartBERT as the first model for AFC against chart evidence. ChartBERT leverages textual, structural and visual information of charts to determine the veracity of textual claims. For evaluation, we create ChartFC, a new dataset of 15,886 charts. We systematically evaluate 75 different vision-language (VL) baselines and show that ChartBERT outperforms VL models, achieving 63.8% accuracy. Our results suggest that the task is complex yet feasible, with many challenges ahead.

## introduction charts are often used to present data in news articles, reports, scientific publications, and across social media posts @xcite . for example, in recent years, charts have been widely used to guide policymakers in deciding health policies and to communicate covid information with the general public; a popular example is the coronavirus dashboard by johns hopkins university, 1 which was integrated in several websites @xcite . misinformation can spread through charts in various ways. previous works in data visualization have discussed how misleading chart design can cause misinformation @xcite . however, a more subtle form of misinformation occurs during chart interpretation (e.g. through invalid comparisons, framing correlation as causation, or spreading of misleading claims). to identify these misinformation types not only the stand-alone chart but the chart together with its message need to be 1 https://coronavirus.jhu.edu/map.html claim: both thane baker and nate cartmell were ranked last. evidence: label: supports figure 1 : an example from the chartfc dataset where the claim is supported by the evidence chart. considered jointly @xcite . in this work, we focus on verifying whether charts support or refute claims about them. there has been substantial progress in automated fact-checking (afc) in recent years, with a focus on verifying claims against text @xcite @xcite @xcite , table @xcite @xcite , and image @xcite @xcite evidence. previous work has widely ignored claim verification against chart images. there are several challenges related to chart fact-checking as opposed to other evidence modalities: the structural information, text in charts, and location of text need to be considered jointly for chart understanding. text plays a key role and is used, for example, as bar labels, chart titles, or in legends to explain the use of colors. moreover, verifying claims against charts requires different reasoning types, e.g. retrieving values, finding extremes, or calculating a sum. to address these challenges, we propose the chart fact-checking task where, given a text claim and a chart, the goal is to classify if it supports or refutes the claim. we introduce chartbert as the first model for afc against chart evidence comprising (i) an ocr-based reading component to extract text and structural information from chart images; (ii) a sequence generation component to process the extracted information; and (iii) an encoding component that extends the bert architecture @xcite with three additional structural embeddings to jointly learn textual and structural representations of chart images. moreover, we release chartfc as the first benchmark for chart-based afc, created using tabfact @xcite as a seed dataset. our dataset contains 15.9k human-written claims and bars of different colors, orientations, and backgrounds (see figure 1 for an example). our highest-performing chartbert model achieves 63.8% accuracy on chartfc. we compare chartbert to 75 visionlanguage (vl) baselines, combining five vision encoders, three language encoders, and five fusion methods. the best-performing vl model is a transformer-based @xcite , dual encoder architecture that uses a simple, yet effective fusion block: concatenation and gated recurrent units (grus) @xcite . our results suggest that state-of-the-art vl approaches struggle with the proposed task, calling for more research. our contributions are as follows: 1) we propose the chart fact-checking task and build chartbert as the first chart fact-checking model; 2) we introduce chartfc, the first dataset for afc with chart evidence; 3) we systematically evaluate state-ofthe-art language/vision encoders and fusion methods on the proposed task, highlighting challenges and providing an analysis of common reasoning types that contribute to failures. foot_0 2 related work ## chartbert model we introduce chartbert, a first bert-based chart fact-checking model. our model consists of (i) a reading component which extracts text and structural information from charts (section 3.2); (ii) a component for generating textual sequences from the information previously extracted (section 3.3); and (iii) a bert-based encoder with additional structural embeddings for the text extracted from charts (section 3.4). the model architecture is shown in figure 2 . ## conclusion and future work we propose the chart fact-checking task and introduce chartbert, a novel model for fact-checking claims against chart images comprising three main components: a reading component, a sequence generation component, and an encoder that extends bert's encoder with structural embeddings. we also introduce chartfc as the first dataset for fact-checking against chart images, consisting of 15, 886 claims and chart images. chartbert achieves 63.8% accuracy on chartfc. we systematically evaluate 75 different vl baselines, using various language encoders, vision encoders, and fusion methods. the highestperforming vl baseline uses bert as language encoder, resnet to extract image representations, and concatenation to obtain joint representations for both modalities. the model achieves 62.7% test accuracy. our results indicate that chart factchecking, which requires extracting and reasoning over text and structural information from charts, is a challenging task for future research on afc and vl methods.