Enhanced Chart Understanding via Visual Language Pre-training on Plot Table Pairs (2023)

Abstract: Building cross-model intelligence that can understand charts and communicate the salient information hidden behind them is an appealing challenge in the vision and language (V+L) community. The capability to uncover the underlined table data of chart figures is a critical key to automatic chart understanding. We introduce ChartT5, a V+L model that learns how to interpret table information from chart images via cross-modal pre-training on plot table pairs. Specifically, we propose two novel pre-training objectives: Masked Header Prediction (MHP) and Masked Value Prediction (MVP) to facilitate the model with different skills to interpret the table information. We have conducted extensive experiments on chart question answering and chart summarization to verify the effectiveness of the proposed pre-training strategies. In particular, on the ChartQA benchmark, our ChartT5 outperforms the state-of-the-art non-pretraining methods by over 8% performance gains.

## introduction chart figures serve as the visual summary of tabular data, which helps to convey rich context in various documents, such as scientific papers, textbooks, and technical news. an intelligent agent that can understand and communicate chart plots can lead to many useful applications. for example, a virtual doctor who knows how to answer the patient's question on a complex medical report or a reading assistant who can summarize the key findings from scientific papers in brief language. in the past few years, there has been a growing interest in our community to explore chart understanding in vision and language (v+l) tasks and many related benchmarks like chart question answering (cqa) @xcite @xcite and chart summarization (cs) @xcite are introduced. while prevalent in the research community, automatic chart understanding remains a challenging problem due to its complex compositions of various shapes, lines, colors, and scene text. although tremendous success is achieved in the v+l research, applying these existing methods to handle chart-related tasks is hard. recent research chartqa @xcite and @xcite attempt to first convert chart images to their underlined tables and use the extracted tables to perform chart-related v+l task. as the extracted tables always have clean and organized structures, it makes extracting relevant information to solve downstream reasoning tasks much more accessible. empirically, using tables yields promising results on both cqa and cs. despite valuing table as a significant ingredient for chart understanding, we have two main concerns about this approach: (1) automatic table extraction is unreliable. existing methods @xcite are often limited to work on a few particular types of chart images and do not generalize well. moreover, the extracted table is likely to contain incorrect noisy predictions that potentially harm the performance of the following task. (2) in most cases, the whole table is optional for resolving the chart-related v+l task. as illus-trated in fig 1 , to answer the question "what is the value of india bar", the model just needs access to the second row to give the correct answer. in contrast, having redundant table information makes finding the relevant information challenging. to better leverage the table data, we argue that it is important to equip the v+l model with the capability to dynamically interpret the table value from the chart information. therefore, in this paper, we propose chartt5, an ocr-based image-to-text generation model pretrained on a self-collected chart table pairs corpus. more specifically, chartt5 learns how to uncover a masked table with two proposed pre-training objectives: masked header prediction (mhp), and masked value prediction (mvp). mhp helps improve the model's capability of linking scene text to the corresponding table headers. mvp requires the model to perform mathematical reasoning over chart structure units and the scene text to predict the correct data value. we evaluate our chartt5 on two tasks and benchmarks: chartqa and chart-to-text. in chartqa, chartt5 outperforms all the non-pretraining methods that use extracted tables by at least 8% performance gains. chartt5 also beats the pre-training table-based methods, which demonstrates the effectiveness of the proposed pre-training strategies. on chart-to-text, chartt5 consistly outperforms the existing sota on the content selection metrics @xcite which values the model's capability to extract the critical information from the chart. in summary, our contributions are summarized below: ## method in this section, we first introduce the dataset for pre-training. we then go over our chartt5 model architecture and pre-training objectives to predict masked tables from the chart and ocr information. ## experiment in this section, we detailed our experiment setups to evaluate the proposed chartt5 on two tasks: chart question answering and chart summarization. we then introduce the main results of the two evaluation tasks. finally, we present the ablation study on chart-table pre-training and the two pre-training objectives. ## conclusion we propose chartt5 to enhance the vision language model's ability to understand chart images via chart-table pre-training. the model learns to interpret the masked tables via our proposed masked header prediction and masked value prediction objectives. chartt5 achieves significant improvement over table-based non-pretraining sota methods on the chartqa dataset, especially on the extractive question sets. we also achieve a new sota content selection score on the chart-to-text summarization dataset. we conduct comprehensive ablation studies to identify the impact of chart-table pre-training, and we find that the proposed pretraining is extremely helpful to extract accurate information from the chart. for future research directions, we believe it may also be meaningful to explore chart understanding under data-efficient settings @xcite and for evidence retrieval tasks @xcite .