Chart-to-Text: Generating Natural Language Descriptions for Charts by Adapting the Transformer Model (2020)

Abstract: Information visualizations such as bar charts and line charts are very popular for exploring data and communicating insights. Interpreting and making sense of such visualizations can be challenging for some people, such as those who are visually impaired or have low visualization literacy. In this work, we introduce a new dataset and present a neural model for automatically generating natural language summaries for charts. The generated summaries provide an interpretation of the chart and convey the key insights found within that chart. Our neural model is developed by extending the state-of-the-art model for the data-to-text generation task, which utilizes a transformer-based encoder-decoder architecture. We found that our approach outperforms the base model on a content selection metric by a wide margin (55.42% vs. 8.49%) and generates more informative, concise, and coherent summaries.

## introduction information visualizations such as bar charts and line charts are commonly utilized by people to get insights from data and make informed decisions. however, understanding and getting insights from charts can be difficult and time-consuming. this is because people often need to visually compare between several graphical marks (e.g. bars) of a chart to infer key insights from data, which may be challenging when the chart involves many data items @xcite . generating a natural language summary to explain a chart has numerous benefits and potential applications. it can help users in understanding and interpreting charts by conveying key points about the chart by focusing on temporal, causal, and evaluative aspects @xcite . it can help people identify insights from charts that they otherwise might have missed. a previous study conducted on a chart corpus found that the text associated with the chart failed to convey any insight from that chart in 35% of the instances, while in another 26% of cases the text conveyed only a portion of the chart's intended message @xcite . therefore, effective chart summarization could help data analysts, business analysts, or journalists in better preparing reports from data. such summaries could also enable people who are visually impaired or have low cognitive abilities to comprehend charts and perform analytical tasks through audio @xcite . however, natural language generation (nlg) systems for charts are still in their infancy. early work mostly focused on statistical methods for finding salient information from charts and planningbased approaches for structuring content @xcite to generate textual captions from basic charts @xcite @xcite . commercial systems such as quill (quill, 2020) and wordsmith @xcite have similarly adopted statistical algorithms and template-based nlg methods which are used to produce data facts in textual form. unfortunately, the predefined template-based nlg methods and planning-based architecture for generating summaries often lack generality and may not offer variations in style. moving beyond template and planning-based approaches @xcite , more recently researchers considered data-driven neural models for generating text from data tables @xcite . however, they do are not designed for chart summarization as they do not consider the chart specific features (e.g. chart type). to our knowledge, there have not been any efforts to develop deep neural models that are specifically tailored for generating chart summaries. in this paper, we present a neural model for chart summarization by extending a transformer-based figure 1 : an example of a generated natural language explanation from a chart from our model. model that was originally designed for the datato-text generation task. a key challenge in developing neural models for chart summarization is the lack of a suitable dataset containing a large set of charts and corresponding human-written summaries. to address the challenge we first developed a corpus of 8,305 charts, where for each chart we crawled the corresponding data table and the human-written summary. our model learns content selection and text generation from such a collection of chart-summary pairs to generate summaries of charts. we also introduce a data variable substitution method that replaces the tokens referring to chart data with variables to generate more factually correct statements than a competitive baseline. as shown in figure 1 , the resulting summary is quite informative, concise, and coherent. the contributions of our paper are three-fold: to our knowledge, our work is the first to investigate the problem of chart summarization using a data-driven deep neural model. ## chart summarization dataset there have been several benchmark datasets that are made available recently for the data-to-text generation tasks @xcite @xcite . however, to the best of our knowledge, there are no publicly available large datasets of chart data paired with human-generated summaries. while some prior work on generating captions and summaries from charts (e.g. @xcite @xcite ) exist, to our knowledge they do not provide any large datasets with chartsummary pairs. with the above challenges in mind, we create a new dataset for chart summarization which is available at https://github.com/jasonobeid/ chart2text . in order to find the best source of data for our corpus, we have analyzed publicly available charts from various sources such as textbooks, research papers, news articles, and websites that contain data charts and facts. this led us to choose statista foot_3 as our data source. statista regularly publishes charts from data collected by market and opinion research institutes, and data derived from the economic sector. since statista has all the necessary metadata including the data tables, titles, and concise summaries of charts it was a suitable source for our purpose. the dataset was crawled from 23,382 freely accessible pages from statista.com in early march of 2020, yielding a total of 8,305 charts, and associated summaries. for each chart, we downloaded the chart image, the underlying data table, the title, the axis labels, and a human-written summary describing the statistic. after examining the crawled dataset, we found that out of 8,305 charts, 7,726 charts were lacking the x-axis labels. to address this problem we searched by regular expressions and applied named entity recognition using corenlp @xcite on the data table to automatically identify if the x-axis represented common temporal dimensions such as years or months. for the remaining 1,353 missing labels we used human annotators who examined the title, summary and the chart to the resulting dataset can be described as follows: let the set of chart-summaries @xmath0, d, t, l, s], where for each chart-summary in cs there is a chart image @xmath1, data table @xmath2, title @xmath3, axis labels @xmath4, and summary @xmath5. the dataset consists of both bar and line charts ( the summaries of the charts are concise, with an average sentence count of 5.2, and an average token count of 113.4. the data tables were moderate in size with an average of 32.3 cells. the discourse structures of these summaries were usually similar. they usually start by describing the chart at a high-level in terms of what this chart is about and then describing and comparing salient points in the chart. some common salient information and statistics are extremes (e.g. highest/lowest values) or trends (upward/downward tendencies) or simple value retrieval (e.g. mentioning the first/last data point's value). ## the chart-to-text model in this section, we first describe the transformerbased model for data-to-text @xcite which we use as our base model, followed by our adaptations to this model for generating chart summaries. then, we describe our experiments with model parameters and the training procedure. ## evaluation we first perform an automatic evaluation to approximately measure the performance of our approach compared to the base model @xcite and then we conduct a human evaluation. finally, we perform some qualitative analysis to have a better summarization method evaluation method our model gong et al. bleu score 18.54 17.06 content selection 55.42 (24.06) 8.49 (9.99) ## conclusion and future work in this paper, we tackle the challenge of automatic chart summarization by introducing a new dataset and proposing a neural approach based on the transformer architecture. our approach learns how to generate natural language descriptions from charts by modifying the gold summaries to replace references to chart data values with data variables. as a result, the model learns how to summarize in a more generalized way, and generates more factually correct statements compared to the base model. our model also generates summaries that are more informative, concise, and coherent according to human evaluations. we hope that our work will motivate researchers to further improve the quality of summaries automatically generated from charts, which is a highly under-explored research area. in the future, we would like to develop larger datasets that cover more diverse domains and additional chart types (e.g. pie charts, scatterplots, heatmaps etc.) to further improve the quality and generalizability of our model. also, while we compared our method with a strong baseline from @xcite , we will perform further comparisons with other models which were developed for the data-to-text generation problem. finally, we would like to build applications such as an interactive chart summarization system with a focus on enhancing the accessibility of charts so that blind and visually impaired people can comprehend charts via audio.