U ni C hart: A Universal Vision-language Pretrained Model for Chart Comprehension and Reasoning (2023)

Abstract: Charts are widely used for data analysis, providing visual representations and insights into complex data. To facilitate chart-based data analysis using natural language, several downstream tasks have been introduced recently such as chart question answering and chart summarization. However, existing methods for these tasks often rely on pretraining on language or vision-language tasks, neglecting the explicit modeling of chart structures (e.g., how chart elements are related to each other). To address this, we first build a large corpus of charts covering diverse topics and visual styles. We then present UniChart, a pretrained model for chart comprehension and reasoning. UniChart encodes the relevant text, data, and visual elements of charts and then uses a chart-grounded text decoder for text generation. We propose several chart-specific pretraining tasks that include: (i) low-level tasks to extract the visual elements (e.g., bars, lines) and data from charts, and (ii) high-level tasks to acquire chart understanding and reasoning skills. Our experiments demonstrate that pretraining UniChart on a large corpus with chart-specific objectives, followed by fine-tuning, yields state-of-the-art performance on four downstream tasks. Moreover, our model exhibits superior generalizability to unseen chart corpus, surpassing previous approaches that lack chart-specific objectives and utilize limited chart resources.

## introduction information visualizations such as bar charts and line charts are commonly used for analyzing data, inferring key insights and making informed decisions @xcite . however, understanding important patterns and trends from charts and answering complex questions about them can be cognitively taxing. thus, to facilitate users in analyzing charts, several downstream nlp tasks over charts have been proposed recently, including chart question answering @xcite @xcite , natural language generation for visualizations @xcite and automatic data story generation @xcite . a dominant strategy to tackle these downstream tasks is to utilize pretrained models @xcite @xcite trained on langauge and vision tasks @xcite . however, although effective, such models may not be optimal for chart-specific tasks because they are trained on large text corpus and/or image-text pairs without any specific focus on chart comprehension. in reality, charts differ from natural images in that they visually communicate the data using graphical marks (e.g., bars, lines) and text (e.g., titles, labels, legends). readers can discover important patterns, trends, and outliers from such visual representation @xcite . existing pretrained models do not consider such unique structures and communicative goals of charts. for instance, pix2struct @xcite is a pretrained image-to-text model designed for situated language understanding. its pretraining objective focuses on screenshot parsing based on html codes of webpages, with a primary emphasis on layout understanding rather than reasoning over the visual elements. matcha @xcite extends pix2struct by incorporating math reasoning and chart data extraction tasks, but it still lacks training objectives for text generation from charts and it was trained on a limited number of charts. in this work, we present unichart, a pretrained model designed specifically for chart comprehension and reasoning. unichart is pretrained on a large corpus of charts and it aims to serve as a universal model for various chart-related downstream tasks (fig. 1 ). inspired by the model architecture from @xcite , unichart consists of two modules: (1) a chart encoder, which takes the chart the trend over the years shows a decline in birth rates for all income levels, with the highest birth rate in 2005 for women below the poverty level at 95, and the lowest birth rate in 2019 for women with an income of 200 percent or more of the poverty level at 44. ## chart pretraining corpus to build a large and diverse corpus with various styles, topics, and storage formats, we crawled charts from various online sources. additionally, we utilized publicly available chart datasets suitable for pretraining. the collected charts can be categorized into two types: charts with underlying data tables and charts without data tables. ## method we propose unichart, a unified pretrained model for chart comprehension and reasoning. this section first introduces the unichart architecture followed by its pretraining objectives. ## conclusion we present unichart, a general purpose pretrained model designed for a broad range of chart-related tasks. our model incorporates chart-specific pretraining tasks and is trained on a large and diverse collection of charts and corresponding summaries collected opportunistically using llms. we conducted both human and chatgpt evaluations to show the superiority of our method. while our model sets the state-of-the-art record on four different downstream tasks and showed improved time and memory efficiency, the evaluation also reveals opportunities for improvement. we believe that our model and pretraining data will be valuable resources for future research and encourage further exploration in this relatively new area.