Chart-to-Text: A Large-Scale Benchmark for Chart Summarization (2022)

Abstract: Charts are commonly used for exploring data and communicating insights. Generating natural language summaries from charts can be very helpful for people in inferring key insights that would otherwise require a lot of cognitive and perceptual efforts. We present Chart-to-text, a large-scale benchmark with two datasets and a total of 44,096 charts covering a wide range of topics and chart types. We explain the dataset construction process and analyze the datasets. We also introduce a number of state-of-the-art neural models as baselines that utilize image captioning and data-to-text generation techniques to tackle two problem variations: one assumes the underlying data table of the chart is available while the other needs to extract data from chart images. Our analysis with automatic and human evaluation shows that while our best models usually generate fluent summaries and yield reasonable BLEU scores, they also suffer from hallucinations and factual errors as well as difficulties in correctly explaining complex patterns and trends in charts.

## introduction data visualizations such as bar charts, line charts, and pie charts are very popular for presenting quantitative data. often people use such charts to get important insights from data and make informed decisions. however, it is well-known that inferring key insights from the charts can be quite challenging and time-consuming, as it may require a lot of cognitive and perceptual efforts @xcite . automatic chart summarization is a task where the goal is to explain a chart and summarize key takeaways from it in natural language. chart summarization has several key benefits and potential applications. first, chart summaries can help people identify key insights from charts that they might gold: in 2019, singapore imported approximately 236.8 billion singapore dollars worth of machinery and equipment, making it the country's largest import commodity by value. this was followed by the import of mineral fuels and lubricants, valued at 102.7 billion singapore dollars. tab-t5: machinery and equipment was the most valuable commodity for singapore in 2019, with an import value of 236.8 billion singapore dollars. mineral fuels and lubricants were the second most valuable commodity for singapore, with an import value of 102.7 billion singapore dollars. have missed otherwise. in a study on a chart corpus, @xcite found that chart authors often failed to convey key insights from charts in their corresponding textual captions. thus, automatic summarization could help authors write effective reports and articles on data facts by suggesting explanatory texts. similarly, readers could benefit from such summaries, as studies have found that captions help readers find important points by explaining visually prominent features in charts @xcite . chart summarization offers another important benefit of making charts more accessible to people who are visually impaired since they can use screen readers to understand what is being presented in the chart @xcite . finally, the generated summaries can be leveraged for indexing documents containing charts to improve information retrieval algorithms @xcite . despite its numerous benefits and applications, the chart summarization problem has not received much attention in the nlp community. early approaches relied on template-based text generation methods that combine statistical techniques and planning-based architecture @xcite to generate captions from bar and line charts @xcite @xcite . recently, researchers considered data-driven neural models for describing tabular data @xcite . however, compared to tables, charts serve a different communication goal, and so is the chart-to-text problem. unlike tables which simply list raw data, charts create visual representation of data that can draw a reader's attention to various prominent features such as trends and outliers @xcite . for example, a line chart may depict an important trend whereas a scatterplot may visually communicate correlations and outliers. existing table-to-text approaches are not designed to explain such visually salient chart features in summaries. there are two main impediments to addressing the chart summarization task. first, the lack of large-scale datasets makes it difficult to solve the task using data-driven neural models. second, there are no strong baselines that utilize the latest advances in neural text generation tasks. @xcite made an initial attempt to address this problem with a dataset and a model that utilizes a transformer @xcite architecture. however, their dataset was built by collecting a small set of charts (8,305) from a single source covering only two types of charts (bar and line). also, their approach does not exploit the recent advances in large-scale language model pretraining, which has been shown to be very beneficial for many vision and language tasks @xcite . to our knowledge, there is no large-scale benchmark with a wider range of topics from multiple sources, covering many different chart types, and with models that employ large-scale pretraining. in this work, we present a large-scale benchmark for chart-to-text with two datasets consisting of 44,096 charts covering a broad range of topics and a variety of chart types. we introduce two variations of the problem. the first variation assumes that the underlying data table of a chart is available, while the other introduces a more challenging and realistic scenario by assuming that the chart is in image format and the underlying table is not available. these two problem scenarios motivated us to adapt a variety of state-of-the-art models that combine computer vision and natural language generation techniques as strong baselines; see fig. 1 for a sample model output. our primary contributions are: (i) a new largescale benchmark covering a wide range of topics and chart types; (ii) a set of state-of-the-art neural models which can act as a starting point for other researchers to expand and improve upon; and (iii) a series of automatic and human evaluations as well as in-depth qualitative analysis to identify further challenges. our code and benchmark datasets are publicly available at https://github.com/visnlp/chart-to-text . ## related work chart summarization early work @xcite followed a planningbased architecture @xcite and used templates to generate texts. these systems only describe how to read the chart rather than explain key insights conveyed by the chart. recently, commercial systems such as quill and wordsmith foot_0 as well as research prototypes, e.g., @xcite and @xcite computed statistics (e.g., extrema, outliers) to present facts from a dataset. @xcite also compute statistics to generates bar chart summaries in a bottom-up manner to simultaneously construct the discourse and sentence structures. recently, @xcite used the resnet @xcite to encode the chart image and an lstm decoder to create the caption. a key limitation of the above bodies of work is that sentences are generated using predefined templates, which may lack generality and offer little variation in terms of reported insights, grammatical styles and lexical choices compared to datadriven models. moving beyond template-based summaries, obeid and hoque (2020) adapted a transformer-based model on a dataset of 8,305 charts, while @xcite in contrast, we focus on the generic chart-to-text problem and train several neural models that combine computer vision and data2text generation. data2text generation data2text models generate a descriptive summary for a table of records. they have been used for various domain-specific tasks such as summarizing sports data @xcite , weather-forecast data @xcite , recipe generation @xcite and biography generation @xcite as well as open-domain tasks @xcite . recent methods have primarily used an lstm-based encoder-decoder architecture @xcite @xcite . @xcite found that transformers @xcite yielded more fluent and coherent outputs compared to their lstm counterparts. others focused on controlling the structure of the summary using a planning approach @xcite as well as generating facts by preforming logical inference over the given table @xcite . image captioning there has been swift progress in image captioning largely due to the availability of large-scale datasets @xcite . @xcite developed an object detection model to summarize objects in images while @xcite utilized texts extracted from images using ocr to generate captions. unlike images with real-world objects and scenes, charts have marks (e.g., bars, lines) that map quantitative data. this makes the chart-to-text problem different from image captioning. ## chart-to-text datasets after searching through various sources including news sites, textbooks, and websites containing data facts, we found two suitable sources with sufficiently large numbers and varieties of charts with textual descriptions as we describe below. ## chart-to-text baseline models problem definition we consider two variations of the chart-to-text problem. in the first variation, we assume that the underlying data table of the chart is available, where the dataset can be represented as a set of 4-element tuples @xmath0, t, m, s⟩ n } |d| @xmath1, t , m and s representing the chart image, data table, metadata and textual summary, respectively. for each cell in the data table t , we have the following information: (i) the string value, (ii) the row and column positions, and (iii) whether it is a header cell or not. the metadata @xmath2, c type , c labels ) consists of the title, type (e.g., bar, line) and axis labels. in the second variation, we assume that the data table is not available which makes the problem more challenging as well as realistic because most charts online are in image format and do not have the underlying data tables. for a given input @xmath3, t, m ⟩ or ⟨c, m ⟩, our goal is to generate a textual description ŝ which is a good summary of the chart according to a set of evaluation measures. we consider three categories of models to tackle the task. the first category is image captioning models, where the task is formulated as generating a textual description for the given chart image. the second category is data-to-text models, which rely on the underlying data tables of the charts to produce the corresponding descriptions. finally, we consider a combination of vision and text models, where the models first extract the text using the craft ocr model @xcite and then train with a data-to-text setup. we present three categories of models below (hyperparameter settings for all the models are provided in appendix a.3). ## conclusion we have presented two large-scale datasets for chart summarization. we also provided several state-of-the-art baselines and measures. our evaluation highlights the promise of these baselines and also reveals several unique challenges for the chart summarization task. we hope that chart-totext will serve as a useful research benchmark for model and metric development and motivate other researchers to explore this relatively new area. let s i be the relevance score for sentence i in the paragraph. let l i be the number of lexical token matches between sentence i and the chart. let n i be the number of numerical token matches, excluding year tokens, between sentence i and the chart. let y i be the number of year token matches between sentence i and the chart. let u i be the number of numerical tokens that appear in sentence i but not in the chart. let c be the number of sentences in the paragraph. let content be the content score of the paragraph. let proximity be the proximity score of the paragraph. let dist be the proximity of the paragraph to the chart. -5 ≤ @xmath4 for example, @xmath5, @xmath6. let rel be the relevance score of the paragraph. @xmath7: a paragraph is relevant if it satisfies the following conditions: since the original chartocr model only outputs the raw data values, we we further extend their approach to output the fully-structured data table as follows. first, we utilize the craft model @xcite to recognize the texts of the detected textual chart elements (x-axis labels, and legend labels). then, we associate the data values with their closest x-axis-label and the data series (e.g. a group of bars or line points) with the legend labels based on the color. for example, in figure 11b , the bars are matched with their closest x-axis-labels ('sunday' and 'daily'). moreover, the values of dark blue bars are associated with '2019' legend-label and the values of light blue bars are associated with americans overwhelmingly support limits on political campaign spending, and most think new laws could effectively reduce the role of money in politics. and there is extensive support for reining in campaign spending: 77% of the public says "there should be limits on the amount of money individuals and organizations" can spend on political campaigns; just 20% say they should be able to spend as much as they want. a somewhat smaller majority (65%) says that new campaign finance laws could be written that would be effective in reducing the role of money in politics, while 31% say any new laws would not be effective. in a recent survey of what americans know about science, we asked people to interpret the chart you see here and tell us what it showed. six-in-ten (63%) identify the best interpretation of this chart as "the more sugar people eat, the more likely they are to get cavities." the statistic shows the distribution of employment in brazil by economic sector from 2010 to 2020. in 2020, 9.12 percent of the employees in brazil were active in the agricultural sector, 19.59 percent in industry and 71.29 percent in the service sector. as of 2019, a third of online users worldwide were aged between 25 and 34 years. website visitors in this age bracket constituted the biggest group of online users worldwide. also, 18 percent of global online users were aged 18 to 24 years. the cost of fossil fuels in the electric power industry can vary depending on the source that is used. in general, fossil fuels cost about 2.50 u.s. dollars per million british thermal units (btu) but can range from 2.02 u.s. dollars per million btu for coal to 9.07 u.s. dollars per million btu for petroleum.