VD ebugger: Harnessing Execution Feedback for Debugging Visual Programs (2024)

Abstract: Visual programs are executable code generated by large language models to address visual reasoning problems. They decompose complex questions into multiple reasoning steps and invoke specialized models for each step to solve the problems. However, these programs are prone to logic errors, with our preliminary evaluation showing that 58% of the total errors are caused by program logic errors. Debugging complex visual programs remains a major bottleneck for visual reasoning. To address this, we introduce **VDebugger**, a novel critic-refiner framework trained to localize and debug visual programs by tracking execution step by step. VDebugger identifies and corrects program errors leveraging detailed execution feedback, improving interpretability and accuracy. The training data is generated through an automated pipeline that injects errors into correct visual programs using a novel mask-best decoding technique. Evaluations on six datasets demonstrate VDebugger’s effectiveness, showing performance improvements of up to 3.2% in downstream task accuracy. Further studies show VDebugger’s ability to generalize to unseen tasks, bringing a notable improvement of 2.3% on the unseen COVR task.

## introduction complex visual reasoning is a crucial yet challenging problem that often requires compositionally synthesizing multiple reasoning steps before drawing the final conclusion. for example, to answer the visual question in figure 1 : "do the skiers wear jackets of the same color?", one must identify all skiers, determine the colors of their jackets, and assess whether the colors are the same. end-to-end vision-language models (vlms) excel at individual tasks such as object detection @xcite and visual instruction following @xcite . however, they struggle to generalize to complex tasks requiring compositional reasoning and inherently lack interpretability @xcite @xcite . to devise a more interpretable and generalizable reasoning process, a recent approach leverages the code generation capabilities of large language models (llms) to generate "visual programs" @xcite . as shown in figure 1 , the visual program decomposes a complex question into a sequence of programmatically executable steps. during execution, the visual program invokes foundational specialist models to perform visual perception and synthesize the results of each reasoning step into the final answer. the inherent compositionality of programs allows this approach to perform compositional reasoning while ensuring generalization and interpretability. nonetheless, program errors become a bottleneck for this approach, accounting for 58% of total errors as shown in our evaluation. following the advancement of llm self-refinement in generaldomain code generation @xcite and llm agents @xcite , recent work leverages zero-shot prompting of llms to debug visual programs based on some given feedback @xcite . however, their feedback typically focuses on limited aspects such as compilation errors. furthermore, the zero-shot prompting technique is less effective for self-critique and self-correction of programs, especially for smaller llms, as shown in recent work @xcite @xcite . in this work, we propose vdebugger, a tool trained to debug visual programs by tracking their execution step by step. as shown in figure 1 , vdebugger takes as input the execution states at each step, including the code being executed and the resulting change of variable values. based on such information, the critic identifies fine-grain program ## related work visual reasoning. the large-scale pre-training of vlms has demonstrated significant success @xcite . when fine-tuned, these models can effectively adapt to specific tasks such as instruction following @xcite , visual question answering @xcite , and object detection @xcite . despite their impressive performance on these individual tasks, vlms still struggle with compositional reasoning that requires composing multiple reasoning steps @xcite @xcite . visual programming addresses this problem @xcite by generating executable programs that decompose the question into multiple reasoning steps and invoke specialized vlms for each step. however, the program errors in the generated code become a bottleneck of this approach. self-debugging and self-refinement. we present a comprehensive comparison between this work method feedback training error unit-test execution states llm self-refinement selfrefine @xcite n/a n/a ✗ ✗ reflexion @xcite n/a n/a ✗ ✗ refiner @xcite n/a n/a per step ✓ general-domain code debugging selfdebug (chen et al., 2023) ✓ ✓ ✗ ✗ jiang et al. (2024) ✓ ✓ ✗ ✓ ldb (zhong et al., 2024) ✓ ✓ per block ✗ visual program debugging stanic et al. (2024) ✓ n/a ✗ ✗ define (gao et al., 2023) ✓ n/a per block ✗ vdebugger (ours) ✓ n/a per step ✓ and existing work for self-debugging and selfrefinement in ## vdebugger framework vdebugger consists of two components, a critic and a refiner. the debugging process is illustrated in alg. 1. starting with an initial program p 0 and its execution feedback, the critic model c detects and localizes potential errors. subsequently, the refiner r corrects these identified errors. this iterative process continues until the critic model c algorithm 1 vdebugger algorithm require: critic c, refiner r, score threshold th, max step t , initial program p0 1: @xmath0: for @xmath1: f @xmath2: score, @xmath3, f b) ▷ identify and localize error 5: if @xmath4: return p 7: end if 8: @xmath5, f b, loc) ▷ refine program 9: @xmath6: end for 11: return p deems the program satisfactory. foot_0 execution feedback. in contrast to previous approaches that focus on execution errors and blockwise execution states @xcite , our objective is to develop a general and comprehensive feedback mechanism that can cover a wider range of errors. drawing inspiration from the stepping debugging strategy foot_1 of human programmers, we track the execution process step by step and document each executed program line, the resulting changes in the intermediate variables, and any errors encountered during execution. this feedback information is fed to vdebugger in text format as in figure 8 in the appendix. ceeds a threshold th (0.5 in this work). the critic classify the program p into either correct or incorrect. concretely, c first generates a correctness token chosen from {t ✓ , t ✗ } representing whether the program is correct or incorrect, so the probability assigned to token t ✓ can serve as score. if the token t ✗ is generated, c further generates the error location loc. here, the location loc is a span within program p defined by its start and end positions, which can be a word, a line, multiple lines, or any continuous segment. refiner. conditioned on the error location loc, refiner r rewrites location loc to fix the program. formally, p @xmath7, execute(p ), loc), where the output program p new only differs with the input program p at location loc. ## training of vdebugger with the critic-refiner framework introduced above, now we design an automated pipeline to collect training data tailored for our framework. our goal is to obtain tuples {(p ✓ , p ✗ , loc)} where in each tuple, the correct program p ✓ and incorrect one p ✗ only differ at location loc. as in figure 2 , our pipeline consists of two steps: (1) generating correct programs, and (2) generating incorrect programs with error locations. ## experiments in this section, we aim to: (1) evaluate the effectiveness of vdebugger by comparing against existing self-debugging methods; (2) analyze the benefits brought by each individual component; and (3) demonstrate its generalization capability by debugging programs generated by other llms and by evaluating on unseen tasks. dataset. we experiment on three forms of tasks including 6 datasets: (1) visual question answering with one image, including gqa dataset @xcite targeting compositional question answering and tallyqa dataset (acharya et al., 2019) targeting counting; (1) visual question answering with multiple images, including nlvrv2 dataset @xcite where each question is accompanied by two images; (3) visual grounding including three variants of refcoco dataset @xcite : the original refcoco dataset, ref-coco+ that disallows location descriptions, and refcocog that involves longer and more complex text descriptions. we report accuracy for question answering tasks and iou for visual grounding tasks. for training data collection, we generate 4 training sets for the gqa, tallyqa, nlvrv2 and ref-coco datasets respectively. we use codellama-7b-python @xcite to generate the initial program pool p (0) , and use codellama-7b-instruct to generate incorrect programs p (1) ✗ with both greedy decoding and mask-best sampling. we collect 9∼14k training data for each dataset and in total 47.7k data. detailed statistics are in evaluated models. we use vipergpt @xcite as our base visual program generator before any debugging. we train vdebugger on each dataset based on codellama-7b-python and codellama-13b-python. we further train a generalized variant on the mix of all datasets denoted as vdebugger w/ gen. during inference, we use a maximum iteration step of @xmath8. we compare our method against two code debugging methods: selfdebug @xcite and ldb @xcite . selfdebug debugs the program based on unit-test feedback. since visual programs do not have unit tests available, we replace it with our execution feedback. ldb uses execution states per program block to iteratively rewrite each block, making it more expensive than our strategy. both selfdebug and ldb relies on zero-shot prompting without any training. ## conclusion vdebugger is a critic-refiner framework fine-tuned to detect, localize, and correct errors in visual pro-grams leveraging fine-grained execution feedback. the training data is collected through an automated pipeline that first generates correct programs and then effectively injects errors using mask-best sampling. experiments on six datasets demonstrate that vdebugger consistently brings improvements, and further studies verifies vdebugger's generalization to unseen tasks. a future direction is to allow the visual program debugger to access visual information in addition to relying on textual information, and to jointly train it with foundation vlms.