V is T ext: A Benchmark for Semantically Rich Chart Captioning (2023)

Abstract: Captions that describe or explain charts help improve recall and comprehension of the depicted data and provide a more accessible medium for people with visual disabilities. However, current approaches for automatically generating such captions struggle to articulate the perceptual or cognitive features that are the hallmark of charts (e.g., complex trends and patterns). In response, we introduce VisText: a dataset of 12,441 pairs of charts and captions that describe the charts’ construction, report key statistics, and identify perceptual and cognitive phenomena. In VisText, a chart is available as three representations: a rasterized image, a backing data table, and a scene graph —a hierarchical representation of a chart’s visual elements akin to a web page’s Document Object Model (DOM). To evaluate the impact of VisText, we fine-tune state-of-the-art language models on our chart captioning task and apply prefix-tuning to produce captions that vary the semantic content they convey. Our models generate coherent, semantically rich captions and perform on par with state-of-the-art chart captioning models across machine translation and text generation metrics. Through qualitative analysis, we identify six broad categories of errors that our models make that can inform future work.

## introduction studies have shown that captions can improve the recall and comprehension of the data that charts depict @xcite . for instance, when a caption emphasizes visually prominent features of a chart, like a peak or a sharply declining trend, readers treat this information as the key takeaway @xcite . moreover, for people with visual disabilities, captions (or equivalent descriptions such as alt text) are often the only means of accessing the presented data. however, as evidenced by numerous guidelines @xcite , producing high-quality * both authors contributed equally to this work. to bridge this gap, several research communities have begun to explore methods for automatically generating chart captions, including using templates and heuristics @xcite , adapting image captioning techniques @xcite , or via data-to-text machine translation @xcite . while promising, these approaches have largely produced captions that either describe a chart's construction (e.g., "the graph is plot between 'number of people' x-axis over 'movie genres' y-axis" @xcite ) or provide statistical summaries (e.g., "machinery and equipment was the most valuable commodity for singapore in 2019" @xcite ). however, these captions do not articulate the perceptual and cognitive features that make charts a distinctive and compelling medium for communicating data (e.g., "prices of big tech corporations seem to fluctuate but nevertheless increase over time" @xcite ). indeed, as @xcite find, both sighted and blind readers strongly prefer captions that express this type of content. to automatically produce such semantically richer captions, we introduce vistext: a benchmark dataset of 12,441 pairs of charts and captions. vistext makes two key extensions over prior approaches. first, vistext offers three representations of charts: a rasterized image and backing data table, as in previous work; and a scene graph, a hierarchical representation akin to a web page's document object model (dom), that presents an attractive midpoint between the affordances of chart-as-image and chart-as-data-table. second, for each chart, vistext provides a synthetically generated caption detailing its construction as well as a crowdsourced caption describing its statistical, perceptual, and cognitive features. these crowdsourced captions represent a substantial increase in data over prior comparable datasets @xcite . to demonstrate the possible uses of the vistext dataset, we train three classes of models -textbased caption models, image-guided captioning models, and semantic prefix-tuning. text-based captioning models fine-tune large language models for vistext's chart captioning task, revealing that both data table and scene graph representations can produce compelling and semantically rich captions. following recent advancements in image-guided translation @xcite , we leverage the additional visual affordances in chart images to develop image-guided chart captioning models. finally, since users often have varying preferences about the type of semantic content in their captions @xcite , we apply semantic prefix-tuning to each of our models, enabling them to output customizable captions. our models generate coherent, semantically rich captions across the vistext charts. evaluating against standard machine translation and text generation metrics reveals that our models consistently output captions that accurately describe the chart's construction, such as its chart type, title, and axis ranges. through qualitative analysis of our model's captions, we find that our model competently outputs semantically rich captions that describe data trends and complex patterns. further, we categorize six common captioning errors that can inform the future development of chart captioning models on the vistext dataset. the vistext dataset and source code are available at: https://github.com/mitvis/ vistext . ## the vistext dataset we designed the vistext dataset in response to two limitations existing datasets present for generating semantically rich chart captions. first, existing datasets represent charts as either rasterized images or as data tables. while useful, these representations trade off perceptual fidelity and chart semantics in mutually exclusive ways -images capture the perceptual and cognitive phenomena that are distinctive to charts (e.g., trends or outliers) but pixels cannot express the rich semantic relationships between chart elements (e.g., estimating plotted data values using axis labels). while the vice-versa is true @xcite , tables also present additional caveats. there is not always a one-to-one relationship between the semantics of a data table and chart (i.e., one data table may be the source for several distinctly different charts). moreover, data tables are rarely published alongside charts; and, automatic data table extraction is error-prone due to the diversity of chart types and visual styles as well as the difficulty of reasoning about visual occlusion @xcite @xcite ). second, if existing datasets provide captions that describe perceptual or cognitive features, these captions comprise only a small portion of the dataset. at best, linecap @xcite offers 3,528 such captions for line charts only, while chart-to-text @xcite estimates that roughly 15% of the sentences in its captions across a variety of chart types express such content. in contrast, vistext provides 12,441 crowdsourced english captions that articulate statistical, perceptual, and cognitive characteristics of bar, line, and area charts. in vistext, charts are available as not only data tables and rasterized images but also as scene graphs. scene graphs are hierarchical representations that better preserve perceptual fidelity and chart semantics, are often the format for publishing web-based charts, and can be recovered from chart images @xcite . ## chart captioning models to demonstrate the affordances of the vistext dataset, we train three classes of models. first, we fine-tune large language models to translate from textual chart representations to natural lan-guage captions. these models evaluate the feasibility and impact of scene-graph models compared to prior data-table approaches @xcite . second, as vistext provides multiple chart representations, we adapt image-guided translation @xcite to develop two multimodal chart captioning models: image-scene-graph and image-data-table . finally, since vistext offers captions at different semantic levels and prior work has shown significant differences in readers' preferences @xcite , we explore prefix-tuned models that selectively output l1, l2/l3, or l1+l2/l3 captions. training details are in appendix d. ## evaluation and results to evaluate the vistext dataset and our chart captioning models, we measure the readability and accuracy of generated captions and their similarity to the vistext target caption. we also qualitatively analyze the descriptiveness of generated l2/l3 captions and categorize common errors. ## discussion we present vistext, a chart captioning dataset of 12,441 charts and semantically rich captions. the vistext charts are represented as a rasterized image, data table, and scene graph to provide diverse and complementary data modalities. using vistext, we fine-tune large language models to generate natural language captions from textual chart representations and integrate image-guided chart captioning to leverage multimodal information. utilizing the varied semantic content in vistext captions, we develop semantic prefix-tuned models that output semantically customized captions to meet diverse user needs. evaluations reveal that our models output precise and semantically descriptive captions, performing on par with state-of-the-art chart captioning models (kantharaj et al., 2022) across machine translation and text generation metrics. looking ahead, while accessibility remains a key domain that would benefit from automated chart captioning, and deploying automated chart captioning models into the field is an exciting prospect, we believe the most promising approach for future work lies in "mixed-initiative" (i.e., human + ai) chart authoring systems. in particular, as we describe in our ethics statement below, chart captioning models are currently prone to make a number of factual inaccuracies which can have severe harmful consequences. on the other hand, by integrating these models into chart authoring systems (e.g., tableau, charticulator, data illustrator, or lyra), chart authors can intervene and make any necessary corrections. indeed, such integration offers exciting opportunities to develop novel interactive methods for verifying generated captions. for instance, models like ours could generate an initial caption (or set of captions) based on the chart currently being authored; as the system has access to all three representations of the chart (the back-ing data table, chart image, and structured scene graph), it might automatically segment the caption into independent "data segments" and interactively link and map them to rows in the table or regions on the chart, akin to kori @xcite .