Self-play through Computational Runtimes improves Chart Reasoning (2025)

Abstract: Vision-language models (VLMs) achieve impressive zero-shot performance on multimodal reasoning tasks. Typically, best reported performance is achieved with a zero- or a few-shot prompt. We observe that asking the model to take other routes of solving the same task, such as through code generation, hurts performance. Furthermore, training sets are typically no longer useful for improving model performance through few-shot learning, due to their use in training. Indeed, we observe that auto-prompting techniques such as DSPy (CITATION), when applied on training sets, do not produce few-shot examples that further improve validation performance. Further, when used in conjunction with program-of-thought, performance becomes even worse.Our work overcomes these limitations by introducing a novel self-play programming interface which leverages the ability of VLMs to first generate code to decompose a complex visual reasoning task in sub-tasks, then use itself, or other models, as a tool to solve decomposed tasks. Our approach enables DSPy to not suffer from performance drops, when applied iteratively on training sets. Furthermore, it outperforms zero-shot baselines on difficult chart reasoning benchmarks. We report the performance of our approach on ChartQA, PlotQA and ChartFC. This enables large models, such as Gemini or GPT to autonomously learn how to use themselves as tools and iteratively improve without the need for additional data.

## introduction the ability of vision-language models (vlms) to understand scientific charts is key to enable automated and efficient data analysis. the diversity and complexity of these charts make this a challenging problem, as evidenced by the emergence * correspondence to: tautis@google.com of several benchmarks such as chartqa @xcite , plotqa @xcite , and chartfc @xcite . most recent breakthroughs have been obtained through pre-training and fine-tuning using carefully constructed data mixtures and scalable model architectures @xcite @xcite . we posit that the benefits for downstream users of these models stem more from the ability and flexibility to perform visual in-context learning (icl) on any given task, rather than the performance on the specific downstream task. not only the task of interest may not be present in the data mixture, but the various training stages may inadvertently degrade performance because of the large number of tasks involved. therefore, an emerging class of approaches leverages visual icl capabilities @xcite for solving such tasks without modifying the base model; an llm orchestrates tools @xcite , writes code @xcite @xcite or a mix of both @xcite @xcite . beyond a certain scale @xcite , icl successfully enables combining image understanding with code generation. our work introduces a technique for performing iterative few-shot example mining, through an approach that improves upon performance of existing methods such as dspy @xcite by extracting additional learning signals from existing training sets. we focus our method development on some of the most difficult benchmarks available today for chart understanding, specifically chartqa @xcite , plotqa @xcite , and chartfc @xcite . we leverage a self-refinement approach @xcite to overcome trivial execution errors and the task metric using the golden labels from the training set @xcite . our method is visualized in figure 1 . the method treats training sets as self-play environments for vision-language models (vlms) to improve their chart reasoning abilities. instead of merely using the training data as a static source of examples for supervised learning or few-shot prompting, the training set becomes an active arena where the model can iteratively learn and refine its problem-solving strategies. it iteratively expands the few-shot examples, thus constructing several few-shot pools (for each initial zero-shot prompt) or mixed-shot pools (combining across types of zero-shot prompts). these pools are then used at inference time. unlike prior work @xcite , this process does not require any human supervision. for powerful vlms, such as gemini @xcite , which both generate code and perform visual reasoning, the training loop resembles self-play @xcite , where the model learns how to best use itself to solve a vqa task. furthermore, it taps into improved reasoning capabilities of models which were also trained on code generation datasets @xcite . the process of replacing the zero-shot prompt with few-shots that matched the training label is similar to that of dspy @xcite . however, we show that, without our method, dspy fails to improve over zero-shot regime, even degrading when program-of-thought is used. in our initialization, we seed the process with multiple initial zeroshot prompts which differ through the type of code generated (e.g., program of thought @xcite , @xcite ). uniquely introduced by our approach, we extend the iterative process for up to 10 iterations and we introduce a novel few-shot selection technique that builds upon the learning signals from the multiple iterations. the computational environment constructed through code generation couples values inferred directly from the image (e.g., values of bars or text labels in a chart) with tool (model)'s inference outputs, through basic arithmetic computations @xcite @xcite . a main challenge to address in this setup is how to automatically choose apis to be used in prompts for invoking individual models (as tools). prior work has shown how llms are capable to perform selfdebugging and self-correction @xcite . we study two types of computational environments. the first one extends program-of-thought @xcite conditioning on multimodal input, while the second type makes use of an indirection api, enabling the model to focus on orchestration and decomposition aspects. the self-play environment surfaces and maintains as few-shots those training examples where the model successfully performed such orchestration. furthermore, it enables a model such as gemini, to be both an orchestrator, by predicting the code which decomposes the problem and leverages tools, and a tool, by being called by the underlying api. by keeping the training step outputs for which results match training labels, we form fewshot pools for each type of zero-shot prompt provided. we provide an example question and model generated solution using our work in figure 2 . our contributions can be summarized as follows: (i) we introduce a simple, yet powerful api, for constructing self-play environments for vlms to reuse existing training sets, (ii) we show the effectiveness of our approach by outperforming zero-shot performance on chartqa, plotqa and chartfc for gemini and gpt without additional data, and (iii) we show that our approach can overcome scaling limitations encountered in prior work, when applied within dspy @xcite . ## related work strong capabilities of recent multimodal models in zero-shot regimes indicate continued improvements on difficult reasoning tasks, particularly in the image understanding domain. yet, it is expected that capabilities may differ by modality, due to specific technical challenges stemming from modalityspecific tokenization @xcite @xcite and availability of mixed-modality pre-training datasets @xcite to learn inter-modal de-pendencies. while impressive results are reported for text modality in many-shot regime @xcite , earlier few-shot results on images flattened more quickly @xcite and recent work on classification tasks @xcite highlighted scaling challenges. our approach would directly be accelerated by further breakthroughs in multimodal many-shot regime. ## method figure 1 presents our approach of constructing a synthetic environment through code generation, iteratively constructing better and better few-shot exemplars. similar to auto-prompting techniques @xcite @xcite , our method starts by bootstrapping examples using zero-shot capabilities of vlms. the instructions describe at a high-level how the program solving the visual question should roughly look like, leveraging instruction-following capabilities @xcite , as well as minimal selfrefinement capabilities @xcite . our method can be described through a seed stage, followed by an offline training stage. finally, at test time, the inference stage solves a given test exemplar by making use of the best matching train exemplars constructed during the training stage. ## experimental setup we first introduce tasks of interest and models we make use of our in our setup in section 4.1. then we report the baseline performance using zero-shot, as well as program-of-thought, followed by our proposed method's performance on validation sets. finally, we report main results in section 4. ## on similarity measures we have briefly investigated as to whether the choice of similarity measure substantially improves over random selection, as shown in @xcite . we have made use of clip image similarity, clip text similarity @xcite , tf-idf text similarity, tf-idf text similarity. we observed that clip similarity sometimes reduced the number of images necessary, but inconsistently across datasets. therefore, we have based on our main results on random selection. we investigate more complex selection measures as an in-depth study in the next section. ## scaling self-play on plotqa to further strengthen our contribution, we scale up our analysis beyond the dspy configuration. we chose to focus this study on plotqa to limit costs. we ablate the choice of large model, adding gpt-4o, scale the number of bootstrapping iterations up to 10 and we report the effects of labeling up to 1000 exemplars per iteration during the training stage using our method. due to the large number of examples available, we chose to select the 8shot at test time using a similarity measure based on the question asked. lastly, we also report how the performance changes when image.answer is implemented using a tool, specifically a fine-tuned version of paligemma-3b @xcite . for strengthening our contribution, we do not make use of an additional predictor based on chain-ofthought, rather directly use the self-play predictor output after code execution. when scaling up to @xmath0, we use up to @xmath1. during every step, the teacher typically labels up to 60% of the training set, roughly 600 examples. out of these, during inference stage, we select 8 examples that best match the question at test time. our findings in paligemma an alternative implementation of the self-play api is for the image.answer method to call an external tool. we fine-tune paligemma @xcite on plotqa. then we replace the original implementation that calls the model with the zero-shot prompt with a call to the fine-tuned model. first, we contrast the tool performance with the zero-shot performance in our method improves significantly over the baseline in both cases, as depicted in figure 4 . gemini 1.5 flash is able to improve regardless of the implementation, but benefits from more predictable and higher quality outputs from specialized tools. we attribute the large improvement over the setup in novelty score training stage has 1000 examples which includes questions of varying difficulty. some questions are simple and are always answered correctly, while others are challenging and only solved occasionally. we argue that examples that are hard to solve contain more novel information. for this, we introduce a novelty score that we assign to an example which has been evaluated repeatedly through the @xmath2. this would be defined as follows: n ovelty e @xmath3, where @xmath4, 2, ..., n and k correct is the number of times an example was correctly solved during the training stage. then, at inference time, we combine the similarity measure score with the novelty measure score introduced here in order to choose whether an example should be used as part of the 8-shots selected at test time. the final similarity distance is defined as we report the result in figure 5 . compute estimates for the tasks used, a typical input example consists of a (image, text) pairs, typically resulting in about 300 tokens per input (e.g. 258 tokens for the image, rest for the text). the output token counts differs when generating code ( 50100 tokens) or directly answering as tool ( 10-20 tokens), from within the code. the key additional cost comes from the number of tool calls, which depends heavily on initial prompting approach. lastly, the costs for fine-tuning a tool such as paligemma @xcite need to be accounted for as well. lastly, we report the results on the sub-sampled test set for plotqa in ## conclusion our work introduces a new recipe through which highly capable models, such as gemini multimodal, can leverage their joint image understanding and code generation capabilities for bootstrapping improved performance. we validate our approach on a set of difficult chart reasoning benchmarks, but note it's wide applicability. we do this by seeding environments with zero-shot prompts that solve a given task in two ways, through program-of-thought or through a self-play api that enables gemini to focus on the high-level reasoning challenge, delegating the low-level problem to itself as a tool. our method iteratively improves performance on visual-question answering training sets, generalizing strongly on validation and test sets after just a few training iterations. improvements over zeroshot baselines are strong across each environment and task combination.