Distill Visual Chart Reasoning Ability from LLM s to MLLM s (2025)

Abstract: Solving complex chart Q&A tasks requires advanced visual reasoning abilities in multimodal large language models (MLLMs), including recognizing key information from visual inputs and conducting reasoning over it. While fine-tuning MLLMs for reasoning is critical, collecting and annotating charts and questions is expensive, hard to scale, and often results in low-quality annotations. To address this, we propose Code-as-Intermediary Translation (CIT), a cost-effective, efficient and scalable data synthesis method for distilling visual reasoning abilities from LLMs to MLLMs. The code serves as an intermediary that translates visual chart representations into textual representations, enabling language models to understand cross-modal information and generate reasoning chains accordingly. In this way, we can employ text-based synthesizing techniques to expand chart-plotting code and generate high-quality Q&A pairs for training models. This produces ReachQA, a dataset containing 3k reasoning-intensive charts and 20k Q&A pairs to enhance both recognition and reasoning abilities of MLLMs. Experiments show that models fine-tuned with ReachQA not only perform well on chart-related tasks but also show performance gains on general reasoning benchmarks.

## introduction multimodal large language models (mllms) have achieved notable progress, particularly in visual recognition tasks @xcite . however, their ability to comprehend complex images like charts in real-world contexts and to address reasoning-intensive questions remains limited compared to humans @xcite @xcite . our analysis of the error distribution in chartqa (figure 1 ) also reveals two main failure modes in current figure 1: error distribution of three baseline models vs. our reachqa-trained versions on chartqa test set @xcite , as judged by gpt-4o. error types are categorized into recognition error, reasoning error, and other errors (question misinterpretation, factual inconsistency or hallucination, and response refusal). mllms: while most errors originate from visual misrecognition, a substantial portion arises from flawed reasoning even when visual elements are correctly identified. this contrasts sharply with human performance @xcite , since we can purposefully identify task-critical information from images and engage in step-by-step reasoning processes. these observations motivate our investigation into bridging this capability gap through the acquisition of human-like reasoning patterns. while distilling expert rationales from humans or stronger models presents a promising pathway for improving reasoning abilities @xcite @xcite , constructing high-quality training data for chart-related tasks is expensive and hard to scale. early approaches typically rely on manual chart collection from online sources, meticulous data filtering and annotation @xcite . recent attempts to automate q&a genera-tion through llms often use data tables as inputs @xcite , which neglect the visual-semantic features of charts. even with the use of mllms @xcite , our preliminary study ( § 2.2) shows they also struggle to produce accurate and challenging data for advanced reasoning skill acquisition. in comparison, we find that when llms process charts in a better textual format-code, they can generate q&a pairs at lower costs and with higher quality. inspired by the concept of intermediary translation @xcite , which refers to using a bridge language to improve translation quality across diverse languages in literary studies, we introduce code-as-intermediary translation (cit). in this method, the code acts as an intermediary, converting chart images into textual representations by faithfully encoding visual-semantic features within itself. this process enables llms to understand cross-modal information more accurately, thereby generating visually complex q&a pairs with high-quality reasoning rationales. furthermore, it facilitates the adoption of text-based instruction augmentation strategies, such as self-instruct @xcite and evol-instruct @xcite , to expand the quantity and enhance the complexity of the synthetic charts. starting with 33 seed codes collected from the matplotlib gallery, we synthesize more chart-plotting codes covering diverse types and topics, and then complicate them to create richer ones. finally, using the synthetic codes as a bridge, we generate charts (via python) and instructions (via llms) in a bi-directional process, ensuring the alignment between modalities. with the cit method, we construct reachqa, a multimodal instruction dataset containing 3, 249 reasoning-intensive charts and 19, 963 q&a pairs, all at a remarkably low cost of just $300. the dataset comprises questions focused on both visual recognition and reasoning, designed to address the dual challenges of current mllms. additionally, we create a manually verified test set to assess models' recognition and reasoning abilities independently. experiments demonstrate that reachqa-trained models achieve substantial performance gains across benchmarks, with llava-next-llama3-8b @xcite improving by over 30% on average, while both types of errors are significantly reduced (figure 1 ). notably, these improvements generalize beyond chart-specific tasks to broader multimodal reasoning tasks like math-vista and math-vision-an outcome previously unattainable with existing chart-focused datasets. finally, we explore reachqa's working mechanism and more features, providing actionable guidelines for building performant multimodal datasets. our contributions are summarized as follows foot_0 : ## methodology building on the findings above, we propose codeas-intermediary translation (cit), a data synthesis method for distilling visual reasoning abilities from llms to mllms, as illustrated in figure 2 . in the following sections, we describe how we synthesize intermediate codes ( § 3.1), generate paired charts and instructions ( § 3.2), ensure data quality ( § 3.3), and ultimately construct our dataset, reachqa. ## conclusion in this work, we delve into the key challenges mllms face in complex chart q&a tasks, highlighting their deficiencies in both recognition and reasoning. building on our analysis of existing datasets and the untapped potential of llms, we propose code-as-intermediary translation (cit) as a novel method for distilling llms' abilities to improve mllms. with code as a bridge between visual and textual modalities, cit enables language models to interpret complex charts more precisely, facilitating the generation of higher-quality q&a pairs. our synthetic dataset, reachqa, demonstrates significant performance improvements across multiple models and benchmarks, with gains extending beyond chart-specific tasks to broader multimodal reasoning. we believe cit offers a promising direction for scalable and costeffective multimodal instruction data synthesis.