M at C ha: Enhancing Visual Language Pretraining with Math Reasoning and Chart Derendering (2023)

Abstract: Visual language data such as plots, charts, and infographics are ubiquitous in the human world. However, state-of-the-art vision-language models do not perform well on these data. We propose MatCha (Math reasoning and Chart derendering pretraining) to enhance visual language models’ capabilities in jointly modeling charts/plots and language data. Specifically, we propose several pretraining tasks that cover plot deconstruction and numerical reasoning which are the key capabilities in visual language modeling. We perform the MatCha pretraining starting from Pix2Struct, a recently proposed image-to-text visual language model. On standard benchmarks such as PlotQA and ChartQA, the MatCha model outperforms state-of-the-art methods by as much as nearly 20%. We also examine how well MatCha pretraining transfers to domains such as screenshots, textbook diagrams, and document figures and observe overall improvement, verifying the usefulness of MatCha pretraining on broader visual language tasks.

## introduction visual language is the system that uses tightly integrated textual and visual elements to convey meaning @xcite . it is ubiquitous in the human world with typical examples being charts, plots and diagrams existing in places such as textbooks, scientific papers web pages and many more. visual language is also highly complex -besides texts, its structural units can include line, shape, color, orientation, scale, angle, space, etc. one needs to recognize patterns from these structural units, and perform spatial grouping and/or alignment to extract information for reasoning. whilst being prevalent and important, there is little research on visual language understanding from the machine learning community. visionlanguage models pretrained on natural images or image-text pairs crawled from the web perform badly on visual language tasks such as chartqa @xcite and plotqa @xcite due to the high complexity of jointly modeling language and symbols (more evidence in experiments). pix2struct @xcite is a recently proposed pretraining strategy for visually-situated language that significantly outperforms standard vision-language models, and also a wide range of ocr-based pipeline approaches. pix2struct designs a novel masked webpage screenshot parsing task and also a variable-resolution input representation for pretraining an image-to-text encodedecoder transformer @xcite . in this work, we use pix2struct as the base model and further pretrain it with chart derendering and math reasoning tasks. we argue that visual language understanding needs two key ingredients: (1) layout understanding (including number extraction and their organizations) and (2) mathematical reasoning. (1) is required to discover the underlying patterns of the image and organize the elements in the image in a logical form. (2) is needed to operate on the elements extracted from (1) and derive meaningful information demanded by a task or query. based on these observations, we propose two complementary pretraining tasks for enhancing visual language understanding: chart derendering and math reasoning. in chart derendering, given a plot/chart, the image-to-text model is required to generate its underlying data table or the code used to render it. the second task is math reasoning pretraining. we pick two numerical reasoning dataset math @xcite and drop @xcite , render the input into images and the image-to-text model needs to decode the answers. year 2017 2018 2019 net orders (a380-800) -2 4 -20 net orders (a380f) 10 -17 -10 deliveries 15 12 8 🍵 matcha -207, -24, -0.2, -0.04, 0.3 26 figure 1: matcha defines two types of pretraining tasks: (1) chart derendering (light blue boxes) and ( 2 ) mathematical reasoning (light red boxes). in chart derendering, given a chart, the model needs to decode its underlying rendering code or data table. in math reasoning, given a math question rendered as an image, the model needs to decode its answer. chart derendering teaches the models layout understanding (including number extraction and their organizations) and math reasoning teaches the models numerical reasoning capabilities. we use a suite of visual language tasks to test the effectiveness of our method. most importantly, we test on chartqa and plotqa which are qa datasets about plots and charts. on both datasets, matcha surpasses even the sota model assuming access to charts' underlying data tables and can beat the prior sota without gold data table by as much as 20%. we also test matcha on chart-to-text summarization tasks and observe clear improvements over pix2struct and achieves sota on chart-to-text (kantharaj et al., 2022) pew split. last but not least, to examine if the matcha pretraining generalizes to datasets beyond the standard plots and charts domain, we also test matcha on four additional domains where pix2struct was evaluated on: documents, illustrations, user interfaces, and natural images (including datasets, such as textbook qa, widget captioning, etc.). we demonstrate consistent improvement on most additional datasets compared with the base model pix2struct. to summarize, our contributions are: (1) proposing a set of effective pretraining tasks for visual language learning (2) demonstrating consistent improvements across all evaluated tasks and sota results on chartqa, plotqa, and chart-to-text sum-marization (statista set) without accessing the gold data tables; (3) verify that matcha pretraining transfers to visual language benchmarks beyond the chart & plot domains and achieve sota across a wide range of datasets beyond the chart domain such as textbook vqa and widget captioning; (4) comprehensive ablation and analyses to understand the effect of each pretraining component and its impact to downstream performance. ## related work vision-language research and a lack of attention on visual language. research on visionand-language has predominately been focusing on natural images. visually-grounded reasoning datasets such as nlvr2 @xcite and marvl @xcite are mostly in the natural image domain. synthesized datasets such as shapes @xcite , nlvr @xcite , and clevr @xcite can be seen as in the visual language domain. however, their visual language systems are significantly simpler than those in the real world such as plots and charts. as a result, information extraction from these synthesized datasets is straightforward. be-sides, the queries in the synthesized datasets are relatively naive and do not require complex reasoning (e.g., questions can usually be on spatial relations or counting objects). consequently, current vision-language models can handle the above mentioned synthesized visual reasoning datasets quite well. however, they do not perform well on real-world visual language datasets where both the information extraction and reasoning becomes much more complex (we will show this in §4). ## method we argue that layout understanding and basic math operation capabilities are the key elements for performing visual language understanding/reasoning. we inject such capabilities to the model by proposing two pretraining tasks: chart derendering ( §3.1) and math reasoning ( §3.2) which we introduce in detail in the following sections. ## experiment we detail our experimental setup in §4.1, introduce the main results in §4.2, and results on additional pix2struct tasks in §4.3. ## analyses and discussions in this section, we first conduct pretraining ablations in §5.1 to understand the usefulness of each pretraining component, then in §5.2 we conduct fine-grained analysis and error analysis to probe matcha' strengths and weaknesses. we conduct two types of ablations. first, we remove a whole type of pretraining datasets. for example, 'no math reasoning' means removing the whole math reasoning component and drops the math and drop datasets. the weights of other datasets in the mixture are proportionally increased. second, we remove an individual dataset within a component. for example, 'no math dataset' means removing just math dataset but keep other datasets in the math reasoning component untouched. in this scenario, we increase the weight of other math datasets (in this case just drop) proportionally to maintain the overall weight of the component in the mixture. to reduce compute used, we train one full matcha model and all its ablated models with 50k steps (the original full matcha is trained for 100k steps). as a result the matcha model performance in ## conclusion we have proposed a pretraining method matcha for visual language tasks. matcha injects chart understanding and reasoning knowledge to an image-to-text transformer model by learning to ( 1 ) predict the underlying data tables and code given chart images and (2) decode the answers of math questions (rendered in the form of images). matcha establishes new sota on 5 out of 6 setups across three chart domain benchmarks covering both qa and summarization tasks. on visual language tasks beyond the chart domain (e.g., textbook qa and docvqa), matcha improves upon pix2struct, indicating that the learned knowledge in matcha pretraining can be transferred outside of the pretraining domain. we conduct comprehensive ablation studies to identify the actual impact of each pretraining component and task and find that chart derendering is essential for extractive questions while math pretraining is important for queries that requires complex reasoning.