A uto C hart: A Dataset for Chart-to-Text Generation Task (2021)

Abstract: The analytical description of charts is an exciting and important research area with many applications in academia and industry. Yet, this challenging task has received limited attention from the computational linguistics research community. This paper proposes AutoChart, a large dataset for the analytical description of charts, which aims to encourage more research into this important area. Specifically, we offer a novel framework that generates the charts and their analytical description automatically. We conducted extensive human and machine evaluation on the generated charts and descriptions and demonstrate that the generated texts are informative, coherent, and relevant to the corresponding charts.

## introduction natural language generation (nlg) is one of the core research areas in artificial intelligence @xcite . recent nlg studies have explored data-to-text generation, where exciting applications such as automated news reporting @xcite were developed to generate text from non-linguistic data automatically. in this paper, we explore the chart-to-text generation problem, where analytical textual descriptions are automatically generated for a given graphical chart. chart-to-text generation has many exciting academic and commercial applications. for instance, preliminary analyses can be generated on charts to aid users in authoring analytical documents. on the accessibility front, automatically generated chart analyses can also support accessibility since text descriptions can be fed into speech-to-text modules and help visually impaired individuals to understand charts. chart-to-text generation could also be applied to aid academic writing. text descriptions of visual elements such as diagrams, charts, and graphs, are among the core academic assignments in linguistics @xcite . for example, the ielts academic writing task 1 (awt1) is an assessment task that elicits written responses on a visual-verbal relationship. the awt1 requires test takers to "describe, summarise, or explain the information in a graph, table, chart, or diagram." figure 1 shows an example of the awt1. chartto-text generation offers the potential to generate large-scale chart analytical description learning examples for students attempting awt1. despite the many benefits and applications of chart-to-text generation, this nlg task has received limited attention from computational linguistics and nlg researchers. among the key factors that hinder the development of this research area is the lack of a large chart description dataset that may facilitate chart description studies. intuitively, one possible solution is to collect and manually anno-tate a chart description. for instance, we will first need to obtain a large dataset of charts and subsequently engage human annotators to write the summary and explanations for these charts. however, such a data collection process is time-consuming and expensive. another approach is to perform large-scale data-crawling to retrieve charts and corresponding human-written summaries from the internet. however, it is challenging to ensure that text summaries correctly describe the chart and have provided adequate details to aid readers in understanding the chart as the charts are retrieved from multiple sources. for instance, in a recent study, @xcite had performed a large-scale data collection of charts and corresponding text descriptions. however, the descriptions of the chart in the dataset contained background knowledge beyond the data illustrated in the chart. in this paper, we aim to address chart analysis data scarcity and quality problems by proposing a novel framework that generates charts and their corresponding high-quality descriptions automatically. the autochart 1 dataset generated by our proposed framework will pioneer new computational linguistic and nlg research area on chart descriptions. for instance, the availability of a large-scale chart description dataset encourages the creation of supervised machine learning and nlp models to interpret the charts and generate relevant text descriptions automatically. we summarize our contributions as follows: ## related work there are very few data-to-text works that investigate chart recognition and understanding. many of these existing works focused on extracting data 1 code: https://gitlab.com/bottle_shop/snlg/chart/autochart from the various types of visual charts using deep learning computer vision and object recognition techniques @xcite @xcite @xcite @xcite . for instance, @xcite proposed an automated system that extracted data points from bar and pie charts to create textual descriptions. however, the generated textual descriptions listed data values extracted from the figures in a static format without any analytical discussion about the charts' overall trends or summary. another line of work have also proposed table -to-text models @xcite , which aims to generate long and goodquality description from structured data formatted in a table. nevertheless, these table-to-text models are designed for specific domains and structured data, and it is challenging to adopt these methods in the chart-to-text task. another related sub-domain of work is the visual-based question and answer (q&a) task. @xcite introduced the figureqa corpus, which consists of over one million questionanswer pairs grounded in over 100,000 visual charts. @xcite extended the work in @xcite and proposed the plotqa corpus, which is a larger dataset with 28.9 million question-answer pairs over 224,377 charts from real-world sources and questions based on crowdsourced question templates. while large datasets have been collected for the visual-based q&a task, these datasets are not applicable to generate analytical chart descriptions as the question-answer pairs are often short and data-specific without any in-depth analysis on the charts. closer to our work, @xcite introduced a new large-scale corpus on chart summarization and proposed a transformer-based chartto-text model. however, the descriptions of the chart in the dataset contained background knowledge beyond the data illustrated in the chart. these " noises" from the beyond-chart-data information may affect the learning of text generation models. another prominent data source, statista, has highquality charts, but corresponding summaries may not be descriptive of the chart. our study addresses the limitations of existing chart-to-text datasets. it extends the existing works on chart recognition and data extraction by offering a novel framework to generate charts and their corresponding analytical descriptions auto- trend generation strategy chart-description pairs data points extraction & synthesis multi-types chart generation chart meta-data extraction chart generation module linguistic rhetorical moves analysis analytical description generation module human-written templates paraphrase templates paraphrase algorithm rhetorical moves synthesis for chart description statistical data from multiple sources matically. to this end, we construct and contribute autochart, a large-scale chart analytical description dataset. ## autochart dataset construction the goal of this study is to construct a dataset of charts with their corresponding analytical descriptions automatically. to this end, we propose a novel framework to construct the autochart dataset and illustrate its construction in figure 2 . we begin by collecting statistical data from multiple sources over the web and create the trend generation strategy. the goal of the strategy is to ensure that the generated charts exhibit some form of temporal trends, which ultimately encourages writers to identify these trends analytically. the proposed framework contains two main generation modules: chart generation and analytical description generation. the statistical data and trend generation strategy guide the automatic generation of charts and their meta-information in the chart generation module. specifically, we generate four types of charts: scatter plots, line charts, vertical and horizontal bar charts. in the analytical description generation module, linguistic researchers are first recruited to write the analytical descriptions for a few charts. the humanwritten descriptions are used as templates for the automatic generation of analytical descriptions. as it is labor-intensive to draft human-written descriptions templates, we expand the number of templates by leveraging open-source algorithms to paraphrase the human-written descriptions. subsequently, we analyze the linguistic rhetorical moves of the human-written and paraphrased templates. the rhetorical move analysis enables us to cate-gorize the rhetorical function types of sentences presented in the analytical description templates. finally, the template sentences annotated with rhetorical moves are strategically sampled and adapted to chart data to generate the analytical description for a given chart. ## dataset evaluation in order to conduct a thorough evaluation on the generated analytical descriptions, similar to many nlg tasks, we assess the generated analytical descriptions using both human and automatic metrics @xcite . ## discussion and conclusion the autochart dataset opens up new research opportunities for the computer vision, computational linguistics, and natural language processing research communities. novel object recognition and deep text generative models can be designed to interpret charts and generate relevant analytical descriptions automatically. the automatic interpretation and generation of analytical chart descriptions have many academic and industrial applications. for instance, generating good-quality analytic chart descriptions can guide students to attempt the ielts awt1. the automated analysis of charts is also a valuable function in existing assisted writing tools. the autochart dataset can support the development and exploration of the supervised chart-to-text methods. we opined that this is the start of an emerging research topic, and many future works could be done. as an extension of this work, we aim to investigate and model more sophisticated linguistic techniques to construct better quality analytical descriptions of charts. we will expand the dataset to include more types of charts, e.g., pie charts, box plots, etc. finally, we will also explore more automatic evaluation methods to assess the quality of the generated analytical descriptions. for example, we can examine and assess the analytical descriptions' logic, reasoning, and fluency. to conclude, we have proposed a novel framework that automatically constructs the autochart dataset, a large chart analytical description dataset. we conducted extensive human and machine evaluation on the generated charts and descriptions and demonstrate that the generated text is informative, coherent and relevant to the corresponding charts. we hope that the autochart can encourage more research in the automatic generation of analytical descriptions of charts.