C hart A ssistant: A Universal Chart Multimodal Language Model via Chart-to-Table Pre-training and Multitask Instruction Tuning (2024)

Abstract: Charts play a vital role in data visualization, understanding data patterns, and informed decision-making. However, their unique combination of graphical elements (e.g., bars, lines) and textual components (e.g., labels, legends) poses challenges for general-purpose multimodal models. While vision-language models trained on chart data excel in comprehension, they struggle with generalization. To address these challenges, we propose ChartAssistant, a chart-based vision-language model for universal chart comprehension and reasoning. ChartAssistant leverages ChartSFT, a comprehensive dataset covering diverse chart-related tasks with basic (e.g. bars and pies) and specialized (e.g. radars, and bubbles) chart types. It undergoes a two-stage training process, starting with pre-training on chart-to-table parsing to align chart and text, followed by multitask instruction-following fine-tuning. This approach enables ChartAssistant to achieve competitive performance across various chart tasks. Experimental results demonstrate significant performance gains over the state-of-the-art UniChart and ChartLlama methods, especially outperforming them on real-world chart data with zero-shot setting. The code and data are available at https://github.com/OpenGVLab/ChartAst.

## introduction people around the world generate a multitude of charts daily, including data visualizations for business reports, market analysis, scientific experiments, and data-driven presentations @xcite @xcite . charts are an effective tool for understanding data patterns, such as the distributional properties depicted in histograms figure 1 : a comparison between previous chart-based models and our proposed chartassistant. chartassistant first aligns the chart and the text by pre-training on the chart-to-table translation task. after performing multitask instruction tuning, it can solve various downstream tasks. and growth trends illustrated in line graphs. developing chart learning methods enables the design of machine analysts with enhanced capabilities to solve various chart-related downstream tasks such as chart question answering (qa) @xcite @xcite and chart summarization @xcite . however, chart comprehension is challenging due to the intricate visual marks (e.g. lines, bars and symbols), implicit numerical information, and complex spatial relationships between elements (e.g. axes and labels). interpreting charts requires specialized knowledge, spatial reasoning, and numerical understanding. the advanced general-purpose multimodal models @xcite @xcite such as llava @xcite , trained on natural images, struggle with chart-related tasks due to the specific complex-ities and relationships unique to charts. although recent multimodal literate models @xcite have achieved impressive results in processing various document-level tasks, they still face difficulties in accurately answering chartrelated questions. in pursuit of universal chart reasoning and comprehension, prior works propose pre-training vision-language models on chart-related tasks as shown in fig. 1( @xcite . for example, both matcha @xcite and @xcite perform multitask instructional tuning on chart data. although these methods exhibit good performance on several chart-related tasks, they require taskspecific fine-tuning. moreover, the existing training data @xcite is deficient in image-text annotations aimed at improving the model's comprehension of visual elements and mathematical reasoning, as well as annotated data from the specialized chart types such as box-plots. due to the above factors, existing chartbased models have poor generalization on various downstream tasks as illustrated in fig. 1(a ). to address these challenges, we propose char-tassistant, a new multimodal model for universal chart comprehension and reasoning. to improve generalization, chartassistant is trained on a large-scale chart-specific instruction-tuning benchmark dubbed chartsft. the training process involves a two-stage pre-training pipeline which employs chart-to-table pre-training to align the chart and its structured text and then perform joint tuning on multiple chart-related tasks as shown in fig. 1(b ). as a result, our chartassistant can achieve good results on various chart-related tasks with a single model. we implement chartassistant with two variants, i.e. chartast-d and chartast-s. chartast-d is built upon donut @xcite , a lightweight (260m parameters) but powerful vision-language model for visual document understanding. while chartast-s is built upon sphinx @xcite , a large (13b parameters) vision-language model for universal multimodal comprehension. inherited from sphinx, our chartast-s obtains enhanced chart representation by dynamic resolution processing and mixed visual encoders. therefore, chartast-s offers increased robustness and usability for chart understanding, demonstrating strong performance in various chart-related tasks. specifically, we first construct chartsft by collecting instruction-following data from various chart-related tasks. to address the limitations of existing chart-based benchmarks @xcite @xcite , we introduce several modifications to improve the quality of data annotation: 1) instruction-following data involving various topics for chart-to-table translation is added, which we find helps align the chart and the associated structured text; 2) the chain-of-thought annotations for chart numerical qa task are generated to improve mathematical reasoning abilities @xcite ; 3) the task of chart referring question answering is created to enhance the understanding of visual elements and their relationships @xcite ; 4) chart with specialized types such as radar and box plot are included to improve the generalization. overall, chartsft encompasses a larger corpus of instruction-following data, incorporates a wider range of chart-related tasks and types, and features more comprehensive data annotations compared to previous benchmarks @xcite @xcite . before conducting multitask instruction tuning, as done in existing research @xcite , we start with pre-training char-tassistant on the chart-to-table translation task as shown in fig. 1(b ). this task involves parsing a chart and generating a markdown table. it shares similarities with dense captioning for natural images, allowing the model to interpret the elements and relationships within the chart. similar to the role of image captioning in training multimodal models @xcite @xcite , chart-to-table translation facilitates alignment between the chart and its structured text. following pre-training, we proceed with multitask instruction tuning using chartsft. this two-stage training approach enables chartassistant (a single model) to achieve strong performance across a range of chart-related tasks. the contributions of this paper can be summarized as follows. 1) we present chartassistant, a vision-language model for chart comprehension and reasoning. chartassistant is versatile enough to solve various chart-related tasks across a wide range of chart types. 2) we build a chart-specific visual instruction-following benchmark dubbed chartsft. chartsft surpasses existing chart-based benchmarks with its larger instruction-following data corpus, a broader range of tasks and chart types, and more comprehensive data annotations. ## chartsft we construct a large-scale chart-specific instruction-tuning benchmark called chartsft by collecting data from various tasks. the composition of chartsft is shown in overall, our chartsft encompasses nine types of charts by collecting data from various sources as shown in ## conclusion our work is aimed at developing a generalized multimodal model for chart-related tasks. we propose chartsft, a comprehensive and expansive dataset with the most diverse range of supported chart tasks and types. in conjunction, we suggest chartassistant, a multimodal model trained using a two-stage strategy over chartsft, which can achieve state-ofthe-art results across multiple chart-related downstream tasks. through detailed experiments, we further demonstrate the superiority of it.