P rop T est: Automatic Property Testing for Improved Visual Programming (2024)

Abstract: Visual Programming has recently emerged as an alternative to end-to-end black-box visual reasoning models. This type of method leverages Large Language Models (LLMs) to generate the source code for an executable computer program that solves a given problem. This strategy has the advantage of offering an interpretable reasoning path and does not require finetuning a model with task-specific data. We propose PropTest, a general strategy that improves visual programming by further using an LLM to generate code that tests for visual properties in an initial round of proposed solutions. Our method generates tests for data-type consistency, output syntax, and semantic properties. PropTest achieves comparable results to state-of-the-art methods while using publicly available LLMs. This is demonstrated across different benchmarks on visual question answering and referring expression comprehension. Particularly, PropTest improves ViperGPT by obtaining 46.1% accuracy (+6.0%) on GQA using Llama3-8B and 59.5% (+8.1%) on RefCOCO+ using CodeLlama-34B.

## introduction visual reasoning tasks often require multi-hop reasoning that goes beyond surface-level observations. this type or reasoning typically involves complex multi-step processes, external knowledge, or understanding of compositional relationships between objects or entities. end-to-end vision and language models based on deep neural networks trained with huge amounts of data are used to tackle these tasks @xcite @xcite @xcite . however, these methods often fail at multi-hop compositional reasoning as they aim to solve a wide array of reasoning tasks in a single forward pass. recent work has proposed visual programming as a principled way to tackle visual reasoning @xcite @xcite . these techniques work by leveraging a large language model (llm) to generate the logic of a program in the form of its source code that can be used to solve the problem. these methods can combine various tools in complex ways and offer interpretability and the opportunity to diagnose failures in their predicted logic. visual programming methods that rely on code generation and program execution to solve a task still rely on end-to-end pre-trained vision language models (vlms) either as tools that can be invoked by the program or as a fallback option when the generated code contains syntax or runtime errors. in other words, if the generated code contains errors, then a default end-to-end vlm is invoked. for these methods to be effective, the generated source code should produce solutions that lead to correct results on average more often than their fallback vlm. however, there are still many instances where a generated source code contains no syntax or runtime errors, but the logic of the program produces results that contain incorrect logic to solve the problem. some of these are easier to spot, such as instances where the code returns the wrong data type, or the wrong type of answer for the given problem (e.g. answering with a location when the question is about a quantity). we posit that code testing and assertion error checking which are established practices in software development, should also help these types of methods in guiding them toward better solutions. we introduce proptest, a visual programming framework that generates automatic property test cases to guide code generation and identify logic that is likely to contain errors. fig. 1 showcases a motivating example for our proposed method. proptest first generates property test cases using an llm which probes for data type inconsistencies, syntactic errors, and semantic properties of the results. for instance, in the showcased question if len(appliance_patches) == 0: return "no" for appliance_patch in appliance_patches: ## method we introduce proptest, a framework for leveraging property test code generation. a commonly recommended practice in software development is to write tests first and then write the code for the logic of the program so that it passes the tests. this is the responsible programmer approach to software development. we emulate this approach in proptest by first generating testing code and then generating question result (1) property test case generation llm python code (3) execution llm property test case (2) code generation result vlms property test case api description question error & & & figure 2: an overview of proptest. given an image and a question, the goal is to generate python code that can be executed to get an answer. proptest first calls an llm to generate test cases based on the inferred properties of the answer. then, the generated test cases are used to improve the quality of python code. let us consider a question such as what kind of toy is the boy playing with?, we can easily infer that the answer should be a type of toy. we utilize this insight to provide information to the code generation model, narrowing down the search space rather than only relying on single-step prompt optimization. additionally, generating property test cases is generally simpler than generating code because test cases are shorter and more straightforward. creating an easier test case first sets a baseline to generate more complex code. property test cases guide the code generation process and increase the likelihood of generating accurate and effective code solutions. our framework first generates property test cases using an llm by providing a problem statement as a prompt, e.g., a question, or a referring expression. the source code for these generated tests is then added to the prompt of the llm, along with the original problem statement and detailed api documentation of the available tools or modules. we employ the same api and tools used in vipergpt @xcite , which also relies on generic functions from the python programming language. the code generation model then outputs the code solution that addresses the problem statement and returns a plausible result. we concatenate the generated property test case and the code solution and apply an execution engine where we also provide the visual input. there can be a syntax or runtime error inside the generated main code. an assertion error will occur if the code output does not pass any of the property test cases. if execution proceeds without errors, including syntax, runtime, or assertion errors, the result is returned, and the process concludes. in the event of an error, we default to a task-specific fallback vlm and return. ## property test case generation the purpose of using a property tests is to verify whether a generated code works as expected and guide an llm to generate better code that meets basic properties. the design of property test cases varies based on the data type of the answer due to the different tools (apis) available for each type. in this section, we explain in detail the design process for prompts used to generate property tests for visual question answering tasks, where the task answer is text (section 3.1) and for visual grounding tasks, where the task answer is an image with bounding boxes (section 3.2). ## experiments we introduce the experimental setup (section 4.1), and results on different llms (section 4.2) ## error analysis & discussion in this section, we first focus on the question: what types of errors does the code generation model produce? we analyze the errors in the generated code from vipergpt and proptest across datasets, categorizing them into three basic python errors: assertion, runtime, and syntax errors. we report results using llama3-8b in we first note that code generation models produce more errors in visual grounding tasks than in vqa tasks. this is because visual grounding involves stricter assertions in test cases, leading to a higher frequency of assertion errors. in visual grounding, all test cases check the result image_patch for specific properties, and errors occur when objects or properties are missing. in contrast, vqa often involves simpler yes-or-no checks, where incorrect results might still pass the test. furthermore, refcoco+ has a higher overall error rate compared to refcoco due to its com- detailed analysis with examples is in appendix c. we also find that due to additional assertion errors, proptest has higher overall errors compared to vipergpt. nevertheless, proptest notably reduces runtime and syntax errors on three datasets (e.g., 322 → 227 runtime, 89 → 39 syntax errors in gqa). this reduction indicates that the inclusion of property test cases enhances code generation quality in the aspects of runtime and syntax errors. however, the increase in assertion errors, leading to a rise in total errors, implies that proptest relies more on the fallback model. this raises the question: does the performance gain of proptest come from an increased dependence on vlms? to address this, we compare the performance of vipergpt and proptest without using the fallback model for error handling, as shown in now, we move on to another question: how does running a test case during execution help when there is an error? to address this, we compare proptest with an approach where property test cases are only provided for code generation but are not executed to catch errors ("w/o running test" in in the presence of errors increases accuracy, indicating that our generated property test cases are effective at detecting incorrect code (e.g., +0.8 in a-okvqa). furthermore, we ask another question: does the proptest improve the quality of the code in cases where the baseline also generates correct output types? to tackle this, we compare the results where both the vipergpt @xcite and proptest produced correct output types. to extract the samples where both the vipergpt and proptest produced correct output types, we run generated property tests on the outputs of the vipergpt. we sampled 1000 subsets from each benchmark and gathered the samples where the output of the code solution passed the property tests in both vipergpt and proptest. we used the code solutions by llama3-8b. since a-okvqa uses soft accuracy as a metric, we assume the output is correct when the soft accuracy is larger than 0.5. we consider the result to be correct if the iou exceeds a threshold of 0.7 for refcoco and refcoco+. the results shown in ## property test analysis in this section, we investigate generated property tests in depth by comparing two types of vqa property test cases (section 6.1) and evaluating the generated property test cases (section 6.2). ## related work end-to-end vision language models (vlms) are generally trained on large datasets containing images paired with text descriptions or instructions @xcite @xcite @xcite @xcite . by learning correlations between visual features and linguistic patterns, vlms can understand sophisticated relations between images and text using a single forward pass through a deep neural network. these models, however large, are still bounded by what functions can be learned and encoded in their model weights. on the other hand, with the rise of llms for code generation in recent years @xcite @xcite @xcite , a recent set of methods in visual recognition have adopted the use of these models to solve visual tasks using a hybrid approach where vlms and other computer vision models are used as tools by one of these code generation llms to generate a program that can solve a given task @xcite @xcite . this type of neuro-symbolic reasoning model was referred to as visual programming by @xcite . these methods lead to an executable program that decomposes complex visual reasoning queries into interpretable steps, which are then executed to produce results. these methods define apis (tools) they use during the execution, with functions mapped to off-the-shelf vision modules such as object detectors @xcite , depth estimators @xcite , among many others. these methods benefit from not needing extra training while enhancing reasoning capabilities and interpretability. the performance of these methods depends on the tools or apis the model leverages and the quality of the generated code. one line of work focuses on creating better and more diverse toolsets to improve accuracy @xcite @xcite . efforts to enhance code quality have been made by code refinement techniques, incorporating various types of feedback, such as visual, textual, error-related, and human feedback @xcite . self-tuning mechanisms have also been explored to optimize model hyperparameters automatically @xcite . training a code debugger to detect and fix the code has been investigated @xcite . our method builds upon these findings, aiming to maximize the efficacy of vlms @xcite through property testing that is more specific to the visual domain. meanwhile, writing test cases is a common technique used by software developers to avoid writing code that contains programming errors. similarly, it has enhanced code generation in code contest tasks. test cases are used to detect errors and give feedback for self-refinement @xcite @xcite ). another line of work generates test cases by mutating existing test inputs @xcite or by using llms @xcite . our research, however, differs from these methods by generating property tests that check different properties of the output, and utilizing these tests as an additional input when generating code. ## conclusion this paper presents proptest, a novel framework for leveraging property test code generation to improve the quality of generated program code in visual programming. proptest shows consistent improvements on vqa and visual grounding with four open-source code generation llms. interestingly, we find that common software development advice which dictates that we should first write testing code before implementing new functionality, also applies to llm-based code generation.