Faithful Chart Summarization with C ha TS -Pi (2024)

Abstract: Chart-to-summary generation can help explore data, communicate insights, and help the visually impaired people. Multi-modal generative models have been used to produce fluent summaries, but they can suffer from factual and perceptual errors. In this work we present CHATS-CRITIC, a reference-free chart summarization metric for scoring faithfulness. CHATS-CRITIC is composed of an image-to-text model to recover the table from a chart, and a tabular entailment model applied to score the summary sentence by sentence. We find that CHATS-CRITIC evaluates the summary quality according to human ratings better than reference-based metrics, either learned or n-gram based, and can be further used to fix candidate summaries by removing not supported sentences. We then introduce CHATS-PI, a chart-to-summary pipeline that leverages CHATS-CRITIC during inference to fix and rank sampled candidates from any chart-summarization model. We evaluate CHATS-PI and CHATS-CRITIC using human raters, establishing state-of-the-art results on two popular chart-to-summary datasets.

## introduction chart summarization requires faithfully extracting quantitative data and describing them using natural language. recent natural language generation (nlg) studies have explored different flavors of chart-to-summary generation tasks including caption generation for scientific figures @xcite , chart summary generation @xcite , or analytical textual descriptions for charts @xcite . these tasks can be advantageous for the visually impaired (benji andrews, 2023) as well as for automating interpreting complex domains such as finance 1 code and demo at hf.co/spaces/chats-pi/chats-pi. input: chart image stage 1: sample candidate summaries from model of choice. stage 3: score summaries with the ratio of kept sentences. summary 1: this data is about the impact of the coronavirus pandemic on the level of revenue. 19.8% of companies saw an increase in fixed costs …. summary 2: in 2020, the coronavirus pandemic had a significant impact on the revenues of companies in poland. more than 36% of companies reported a decrease… summary 3: … stage 2: using chats-critic, drop sentences not supported by chart. this data is about the impact of the coronavirus pandemic on the level of revenue. in 2020, the coronavirus pandemic had a significant impact on the revenues of companies in poland. more than 36% of companies reported a decrease in revenues of more than 50%… … x ✓ ✓ ✓ stage 4: re-rank and output the best summary in 2020, the coronavirus pandemic had a significant impact on the revenues of companies in poland. more than 36% of companies reported a decrease in revenues of more than… this data is about the impact of the coronavirus pandemic on the level of revenue. more than 36% of companies reported a decrease in revenues of more than 50%… data-analysis, news reporting, and scientific domains @xcite . while a wide range of models and techniques have been applied for chart summarization, faithfulness remains a major challenge for the task. specifically, the models often misread details in the charts (due to perceptual mistakes) or miscalculate the aggregations (due to reasoning flaws). to overcome some of these limitations, optical character recognition (ocr) models and object detection systems are usually employed to extract meta-data such as axis, values, titles, legend @xcite . these data are then used as auxiliary inputs to finetune nlg models. nonetheless, these modeling efforts are still limited by two fundamental issues (i) training & evaluation dataset quality and (ii) the reference-based metrics being used for evaluation. as examples, two widely used datasets, chart-to-text @xcite and sc-icap @xcite , are automatically extracted from web articles and academic journals. as a result, the summary references are prone to hallucination, i.e. the reference might contain context that cannot be entailed solely by the chart content. training on this data can encourage the nlg models to improvise/hallucinate. besides, the auto-extracted summaries sometimes emphasize only certain aspects of the chart, missing out critical insights. on the other hand, n-gram based metrics such as bleu @xcite , or learned metrics such as bleurt @xcite rely only on gold references. they are not capable of recognizing unreferenced but correct insights since they solely rely on the reference for scoring the summaries, as shown in figure 3 . this issue is especially pronounced when the gold references are noisy, which is the case for chart-to-text and sci-cap. last but not least, reference-based metrics also heavily penalize summary style mismatches, giving an artificial disadvantage to llms which are not tuned on the task data @xcite . this motivates building a reference-free critic chats-critic (figure 2 ) that can be used as a metric to score and re-rank summaries. we additionally introduce chats-pi (figure 1 ) that leverage chats-critic scores to generate a high quality summaries. we summarize our contributions as follows: ## the chats-critic metric as shown in figure 2 , chats-critic is composed of a chart de-rendering model that generates the table content of the input chart image, and a table entailment model applied on a sentence level. this motivation stems from the observation that fine-grained evaluations are simpler than full-summary evaluations, mirroring the ease observed in human assessments @xcite . ## experimental setup we assess our methods on diverse datasets to prove their broad applicability. scicap @xcite is a large-scale benchmark for figure-captioning. it is extracted from science arxiv papers published between 2010 and 2020 and contains more than 2 million figures. we use 3 subsets of sc-icap: first sentence collection (133k), singlesentence caption (94k data points), and caption with no more than 100 words (131k). tabfact @xcite ) is a large-scale dataset for table-based fact verification. it contains 16k wikipedia tables as evidence for 118k human-annotated statements. this dataset allows us to study fact verification with semi-structured inputs. we use it to evaluate the entailment accuracy of chats-critic. all our models are developed on the dev sets of the mentioned benchmarks and performances are reported on their test sets. we include more detailed descriptions and processing details of the benchmarks in appendix a.1. ## analysis for an overview of the stages refer to figure 1 . we ablated the impact of deplot on chats-critic, using the original tables as a baseline. the findings are detailed in grammaticality defined as the human ratings on grammatical errors (see section 5.1) on non dropped sentences and summaries is reported in ## conclusion in this paper, we tackle the chart-to-summary multimodal task, which has traditionally been challenging since it requires factual extraction and summarization of the insights presented in the image. to measure the quality of a summary (especially faithfulness which has been overlooked by previous metrics), we present a reference-free metric called chats-critic for accurately and factually scoring chart-tosummary generation. chats-critic obtains substantially higher correlations with human ratings compared to prior reference-based metrics. we additionally present chats-pi , a self-critiquing pipeline to improve chart-tosummary generation. chats-pi leverages chats-critic scores to refine the output of any model by dropping unsupported sentences from the generated summaries and selecting the summary that maximizes fluency and chats-critic'scores. compared with state-of-the-art baselines, chats-pi demonstrates stronger summarization quality across the board, achieving better scores for both the chats-critic which stresses faithfulness and also traditional metrics such as bleurt.