Visual Program Distillation with Template-Based Augmentation (2025)

Abstract: Adapting visual programming or prompting large language models (LLMs) to generate executable code for visual tasks like visual question answering (VQA) for specialized tasks or domains remains challenging due to high annotation and inference costs. We propose a low-cost visual program distillation method that can be used for models with at most 1 billion parameters and requires no human-generated program annotations. We achieve this through synthetic data augmentation based on decoupling programs into higher-level skills, called templates, and their corresponding arguments. Experimental results show that, with a relatively small amount of question/answer data, small language models can generate high-quality specialized visual programs with the added benefit of much faster inference.

## introduction visual programming @xcite @xcite refers to generating programs that invoke visual models to solve tasks such as answering questions about images, typically by prompting a very large language model (llm) like gpt @xcite or llama @xcite . visual programming offers greater adaptability and customization for specialized applications such as a personal visual navigation assistant compared to a single vision-language model (vlm). incontext learning with proprietary or closed-source llms can generate correct visual programs for targeted applications but at the cost of long inference time and compute (see figure 1 ). in addition, generating in-context examples (i.e. visual programs written by hand) requires a significant amount of human effort. previous efforts @xcite have made some progress in adapting smaller, opensource llms for dataset-specific visual programs but still suffer from high training and data costs. with our template-based augmentation method, specialized distilled student models achieve comparable performance on answer accuracy with a small percent of question/answer data (≈ 0.1%) and no human program annotations. our goal is to create a specialized visual programming system with two key characteristics: our key insight in achieving such capabilities lies in decoupling the skill or procedure from the question-specific concept. we call the higher-level skills templates and the concepts arguments. for example, the programs "count the red chairs" and "count the green bananas" have the same template,'find(arg1), verify property(arg2), count', but different arguments: 'red' (arg1) and 'chairs' (arg2) ## related works we outline the most relevant related work below. a high-level comparison between our work and others can be found in visual programming a long line of work investigates generating and executing programs to perform visual tasks. early approaches @xcite @xcite generate programs and execute the programs with learned end-to-end neural modules. based on the impressive code generation capabilities of llms @xcite , visual programming frameworks @xcite @xcite generate programs given an api and in-context examples and execute the programs with large pre-trained vision models. visual programming has been applied to many different domains and applications including visual question answering, video question answering, text-to-image generation, and robotics @xcite @xcite . unlike previous visual programming works, we only use small models (≤ 1 billion parameters) to generate programs. one of the challenges that distinguishes visual programming from llm tool use is the dependence of execution and evaluation on the output of underlying models. as noted in other works @xcite , incorrect programs can return (after being executed) the correct answer even if the program is incorrect, which motivates our investigation into different evaluation metrics. a concurrent work, visual unit testing (viu-nit) @xcite , generates visual unit tests for both visual program selection and self-training. viunit also evaluates the unit test quality and program correctness and reports similar differences (36%) between program and answer accuracy. tool-based finetuning compared to promptbased methods, there are relatively few methods focused on improving program generation in visual programming for specific tasks or in the general field of tool-based llms through finetuning. one of the main challenges is the lack of program annotations for input/output pairs. language modeling has a long history designing self-supervised tasks, especially in pre-training @xcite . toolformer @xcite , uses a form of self-supervision, to create a training dataset to finetune an llm on tool-use programs. for each question/answer pair in a pre-existing training dataset, an llm is prompted to generate a corresponding program. if the program decreases the training loss (of the same llm), the program annotation is added to a new dataset. then the same llm is finetuned on the generated dataset. a similar approach is used in chain-of-thought (cot) finetuning @xcite , program correction methods for specific datasets such as vdebugger @xcite and specialized visual programming methods @xcite , such as visrep. lora (?) is frequently used when finetuning llms on the generated data. in visrep, @xcite use a self-training approach similar to toolformer to finetunue an llm for dataset-specific visual programming. selfannotated programs are kept if the executed program returns the correct answer. while there are some similarities between visrep and our work, the underlying setting is different. visrep focuses on improving program generation on an existing llm (7b) through self-training for specialized tasks, while the goal of our work is to distill visual program generation from an llm to a small model (≤ 1b) for specialized tasks. our work also does not require sampling across specific question types or correcting by hand different program annotations during training. we randomly sample the training set and do not write any program annotations or corrections. distillation knowledge distillation @xcite , where a large teacher model annotates unlabeled data that is then used to finetune a smaller, weaker student model, is frequently used across many different applications including image recognition @xcite , masked language modeling @xcite and commonsense knowledge @xcite . one of the difficulties of distillation is the need for a large number of unlabeled training examples which can be expensive to obtain. one way to compensate for this is to train the student model with a multitask objective. the objective consists of a weighted sum of the cross-entropy on the original task and cross-entropy on a closely related part of the task such as the rationale in cot @xcite . codeplan @xcite applies such an idea to code generation where the secondary objective is a natural language version or plan of the code, which has been shown to be effective in prompting-based work @xcite . one downside to multi-objective prompting is that both the teacher and the student models have to generate additional output. in codeplan, during inference, the model first generates the plan and then generates the desired code creating long inference time. such steps might be necessary for complex code generation but are unnecessary for visual programming. instead we use a relatively simple data augmentation method based on abstractions @xcite or higher level plans of existing programs which we refer to as templates. templates do not require any additional forward passes to create and can be augmented using simple techniques such as word replacement @xcite . ## method preliminaries following the notation in vipergpt @xcite , the visual programming objective is to generate a program @xmath0, p) with a program generator π, input query q and prompt p such that when executed with an execution engine ϕ and corresponding visual input x, ϕ(x, z) returns the correct answer. prompt p contains an api and dataset-specific in-context examples. given a teacher model, π t , and a smaller student model (≤ 1 billion parameters) π s , our goal is to distill visual program generation from the teacher to the student for a specific dataset using a minimal number of answer annotations (question/answer pairs). we do not have any human provided program annotations (question/program/answer triplets), but we do know the api used for the teacher model. there are three main steps to our approach: teacher annotation, data augmentation and student training. ## experiments experimental setup for all experiments, the teacher model is gpt-4o-mini. several student models are trained with lora (?): qwen2.5-coder-0.5b @xcite , codet5 @xcite and llama-3.2-1b @xcite . we use the gqa dataset (hudson and manning, 2019) and vqav2 @xcite datasets for our experiments. all student models are trained and evaluated on a single dataset unless indicated by the word 'joint'. we evaluate on the full gqa testdev split and randomly sample 10,000 questions from the vqav2 validation split. gqa is evaluated using exact match and vqav2 is evaluated based on annotator/answer agreement as in the original benchmark. experiments on additional datasets can be found in appendix a. we use a slightly modified api from the original vipergpt paper @xcite . the main differences are some additional functions (to reduce program length) and removal of the use of a vlm if earlier parts of a program fail. please see appendix d for more details. for visual models, we use instructblip (flan-t5 xl) @xcite for general visual queries, owl-vit2 @xcite for detection and clip @xcite for classification. auto-context generation before training the student model, we adapt the teacher to the vqa domain. we investigate the effectiveness of using human-generated programs and programs generated by the teacher model validated by answer correctness. as our aim is to enable visual programming with low cost and effort, we compare using a small number (25) of human-generated programs to generating 1,000 programs (yielding hundreds of validated programs). as shown in evaluation metrics all approaches are evaluated with two metrics: answer accuracy and program accuracy. student models are also evaluated on student/teacher answer agreement or if student and teacher programs return the same answer, regardless of correctness. program accuracy involves a human (the authors) manually evaluating each program (without execution) for correctness. we randomly sample 100 questions from each dataset for program correctness evaluation. the same 100 questions are used to evaluate each method. to ensure the fairness of our program evaluation, we enlist a second annotator to evaluate program correctness on 50 examples: 25 correct and 25 incorrect. the second annotator agreed with the original program evaluation on 47 out of 50 of the examples. program annotations have been released so others can contribute. 1 there are multiple ways to determine if a program is correct. we generally assume a program is correct unless it violates one of these criteria: some examples can be seen in few-shot comparison first, we compare our distilled student models with the most closely related setting: prompt-based visual program generation. other works, such as visrep @xcite , have different objectives and use carefully curated and larger datasets, and thus are not included in our comparisons. furthermore, visrep evaluates on selected subsets of various datasets, and the specific subsets, models, and code have not been made publicly available for comparison. two models, llama-3.2-1b and qwen2.5-coder-0.5b, are evaluated using both few-shot prompting and template-based distillation. codet5 is also evaluated with template-based distillation. there are two versions of distilled qwen2.5-coder-0.5b: one version is trained (and evaluated) on gqa and vqa separately (denoted by 'single') and the other is trained on both datasets (denoted by 'joint'). the same 25 human generated in-context examples (incorporated into the vipergpt @xcite api) used in the auto-context generation experiments above are used per dataset for annotations error example explanation not executable q: is the chair left or right? p: @xmath1, img, ['left','right']) choose_relationship requires a list as input api violation q: what color is the running dog on the left? p: crop_position ('running left',img) "running left" isn't a valid direction/preposition. contradicts question q: what color is the car above the road? p: @xmath2, road) the question states that the car is above the road, not below. does not answer question q: are there two tables? p: @xmath3.find('table ')) the question asks if there are two tables, not how many tables there are. does not include all question information q: is the blue toy small? p: toy.verify_property('small') the question specifically asks about the blue toy. ## limitations and future work there are several limitations and areas for future work: specialization. our method distills a specialized vqa model that is not intended to provide the same breadth of capability as the original llm. our specialized models, however, are much faster and cheaper for inference and easy to produce, making them suitable for targeted application. ## societal implications distilled student models inherit existing biases from llms so care is required when deploying the models in production. there are also privacy risks, so safeguards should be taken to prevent data leakage.