Chart-based Reasoning: Transferring Capabilities from LLM s to VLM s (2024)

Abstract: Vision-language models (VLMs) are achieving increasingly strong performance on multimodal tasks. However, reasoning capabilities remain limited particularly for smaller VLMs, while those of large-language models (LLMs) have seen numerous improvements. We pro-pose a technique to transfer capabilities from LLMs to VLMs. On the recently introduced ChartQA, our method obtains state-of-the-artperformance when applied on the PaLI3-5B VLM by Chen et al. (2023c), while also enabling much better performance on PlotQA and FigureQA.We first improve the chart representation by continuing the pre-training stage using an improved version of the chart-to-table translation task by Liu et al. (2023a). We then propose constructing a 20x larger dataset than the original training set. To improve general reasoning capabilities and improve numerical operations, we synthesize reasoning traces using the table representation of charts. Lastly, our model is fine-tuned using the multitask loss introduced by Hsieh et al. (2023).Our variant ChartPaLI-5B outperforms even 10x larger models such as PaLIX-55B without using an upstream OCR system, while keeping inference time constant compared to the PaLI3-5B baseline. When rationales are further refined with a simple program-of-thought prompt (Chen et al., 2023a), our model outperforms the recently introduced Gemini Ultra and GPT-4V.

## introduction visual language, where text and images work together to deliver information, can be expressed through charts, plots, and diagrams. multimodal reasoning within this context is challenging, as it involves linking visual properties (like color, line style, and positioning) with textual content (such as legends and units). many recent advances of vision-language models (vlms) come from techniques enabling better representations @xcite , giving the model the ability to understand core elements of the image, a necessary building block for basic reasoning. however, complex reasoning capabilities which combine the core representation of the image with semantic understanding of a question to provide an answer, have been rather limited. models oftentimes are not able to contextually combine image and text representations. one technique that improves reasoning capabilities in large-language models (llms) includes in-context learning for eliciting reasoning such as chain-ofthought prompting @xcite , decomposing tasks @xcite or composing stored facts in weights @xcite . fine-tuning on datasets with rationales @xcite has been shown to be effective for smaller models. in this work, we tackle improving reasoning capabilities in vlms through better learned image representations, followed by finetuning on synthetic datasets with reasoning traces generated by more capable llms. we also explore a hybrid online setup for numerical reasoning refinements. we empirically show that this indeed improves performance through experiments on chartqa @xcite . visual-question answering on charts quantifies the ability of a vlm to reason using complex information presented. oftentimes answering the question requires implicit or explicit information extraction, followed by intermediate grouping or computations using the extracted information, and reasoning with the final quantities, as shown in figure 1 . vision-language models (vlms) such as pali-x and pali-3 are hybrid model architectures which use a vision and a language backbone to solve visual tasks @xcite . the training recipe typically involves a pre-training stage fo-question: what's the difference between the highest value of the red line and the lowest value of the green line? answer: 79 cused on learning a good internal representation, followed by a downstream fine-tuning stage. @xcite note that pali-3 falls behind pali-x on chartqa likely due to its limited reasoning capabilities. results presented in this work suggest that the lack of a pre-training task for learning better chart representations, as done in @xcite , may be another reason. enhancing the reasoning capabilities of large language models (llms) such as palm-2 @xcite or @xcite is a very active research area. while reasoning is considered an emerging property with scale @xcite , @xcite argue that simply scaling only enables better memorization of knowledge and does not enable composing multiple stored facts into an answer. on the other hand, prompting techniques enacting complex reasoning on downstream tasks have been shown to be very effective @xcite . transferring reasoning capabilities from large to small models enables reducing serving costs, while increasing task performance. @xcite have introduced an effective multi-task framework which enable small models to outperform their much larger counterparts using less data. they do so by leveraging rationale generation as a separate task, instead of more standard distillation approaches, which first infer the rationale, followed by the answer @xcite . we apply this framework for the first time on multimodal tasks. contributions our main results can be summarized as follows: (i) we introduce an efficient recipe consisting of a pre-training task and fine-tuning task with synthetic datasets using a multi-task setup for improving reasoning capabilities, (ii) we obtain sota performance by significantly improving pali-3 performance on the chartqa benchmark with our recipe and using 10x less parameters than prior work, (iii) we perform numerous ablation experiments quantifying the impact of the techniques used in our recipe. the remainder of this paper is structured as follows. section 2 describes related work, followed by section 3 which introduces the construction of the training datasets. section 4 illustrates our novel pre-training and fine-tuning recipe, followed by section 5 describing the experimental setup and main results. lastly, section 8 delivers a conclusion and recommendation for future work, followed by section 9 where we acknowledge limitations of the current work. ## method our work builds on top of pali-3 architecture and pre-training recipe, which consists of two backbones, a vision transformer vit-2b and text encoder-decoder ul2-3b. our starting point is the recipe described by @xcite . the uni-modal pre-training stage trains the vision encoder using contrastive loss through the siglip loss, while the language encoder-decoder is pretrained using the ul2 loss. both backbones are pretrained jointly using a multi-modal stage. lastly the resolution increase stage enables the vision encoder backbone to work with 812x812 resolution images. we continue pre-training using this checkpoint. ## experiments we describe the general learning hyper-parameters for the pre-training and fine-tuning stages, followed by interpretation of the results. ## performance overview we position our results relative to existing prior work in our work significantly outperforms prior models specialized on the chartqa benchmark. concurrent to our work, chartllama-13b also uses synthetic data generated, but with a fairly different approach. although outside the scope of our work, it may be that the approach took to train the much smaller matcha and unichart models may be combinable with the approach we presented in this work, leading to possible improved performance with even less computational resources. the method introduced in this work can be uniquely combined with much larger models through rationale generation. as shown in the results, rationales generated by vlms can suffice for larger llms to effectively operate on, providing a text-representation of the chart conditioned on the question. our method matches the recently introduced gemini ultra model and outperforms previous approaches. ## future work we highlighted several drawbacks of our approach in section 5.4. the training mixtures do not have examples where colors are used to construct reasoning examples. bootstrapping such examples, for example by running a smaller sized model with questions that extract color related information, then combines them, would likely improve quality. very complex reasoning examples are also limited. specifically, semantically identifying chart elements and performing numeric computations to solve questions would further improve quality. ## conclusion we introduced a novel recipe that significantly improves the reasoning capabilities of vlms. applied to pali-3, our method significantly outperforms even the 10x larger pali-x on the chartqa benchmark, establishing a new state-of-the-art. we demonstrate how the pre-training stage improves downstream performance. our synthetic data generation technique coupled with the use of a multitask setup, successfully transfers reasoning capabilities from larger llms to smaller vlms. moreover, our method enables a computationally more expensive setup where predicted rationales are refined using program-of-thoughts with palm 2-s. the composite solution outperforms gemini ultra and gpt-4v on the chartqa benchmark.