Analyzing Modular Approaches for Visual Question Decomposition (2023)

Abstract: Modular neural networks without additional training have recently been shown to surpass end-to-end neural networks on challenging vision–language tasks. The latest such methods simultaneously introduce LLM-based code generation to build programs and a number of skill-specific, task-oriented modules to execute them. In this paper, we focus on ViperGPT and ask where its additional performance comes from and how much is due to the (state-of-art, end-to-end) BLIP-2 model it subsumes vs. additional symbolic components. To do so, we conduct a controlled study (comparing end-to-end, modular, and prompting-based methods across several VQA benchmarks). We find that ViperGPT’s reported gains over BLIP-2 can be attributed to its selection of task-specific modules, and when we run ViperGPT using a more task-agnostic selection of modules, these gains go away. ViperGPT retains much of its performance if we make prominent alterations to its selection of modules: e.g. removing or retaining only BLIP-2. We also compare ViperGPT against a prompting-based decomposition strategy and find that, on some benchmarks, modular approaches significantly benefit by representing subtasks with natural language, instead of code. Our code is fully available at https://github.com/brown-palm/visual-question-decomposition.

## introduction end-to-end neural networks @xcite have been the predominant solution for visionlanguage tasks, like visual question answering (vqa) @xcite . however, these methods suffer from a lack of interpretability and generalization capabilities. instead, modular (or neurosymbolic) approaches @xcite @xcite have been long suggested as effective solutions which address both of these limitations. these methods synthesize symbolic programs that are easily interpretable and can be executed (leveraging distinct image or language processing modules) to solve the task at hand. the most recent such models @xcite @xcite are training-free: they leverage large language models (llms) to generate programs and subsume powerful neural networks as modules. such approaches demonstrate strong results and outperform end-to-end neural networks on zero-shot vision-language tasks. these recent modular approaches typically include state-of-the-art end-to-end networks, among a complex schema of other modules and engineering designs. as a result, the contribution of these networks is difficult to disentangle from the modularity of their overall system. thus, in this paper, we analyze vipergpt, in which blip-2 @xcite is a constituent module, as a representative example of a recent and performant modular system for vision-language tasks. blip-2 can particularly (and in contrast from vipergpt's other modules) solve vqa tasks on its own. we ask: where does its additional performance come from, and how much is due to the underlying blip-2 model vs. the additional symbolic components? to answer our research questions, we conduct a controlled study, comparing end-to-end, modular, and prompting-based approaches (sec. 2) on several vqa benchmarks (sec. 3). we make the following specific contributions: 1. in section 4, we find that vipergpt's advantage over blip-2 alone is likely due to the task-specific engineering in vipergpt. specifically, we run vipergpt in a task-agnostic setting, in which we do not preselect different subsets of modules for each task (as is done in @xcite ). we find that, without the task-specific module selection, the average gain of vipergpt over blip-2 disappears (dropping from +8.7% to -0.8%). moreover, we find that removing the blip-2 module re- ## models in this section, we share specific design decisions and implementation details for each of the three model families we assess in this paper (i.e. end-toend, modular, and prompting-based). we additionally visualize these approaches in fig. 1 . ## evaluation we evaluate variants of end-to-end, modular, and prompt-based methods (sec. 2) on a set of vqa benchmarks (sec. 3.1) using direct answer and multiple choice evaluation metrics (secs. 3.2 and 3.3). ## selecting modules in vipergpt in @xcite , the choice of modules is different for each task. this contrasts with end-toend models like blip-2 which are purported to be task-agnostic. to draw a more direct comparison, we evaluate vipergpt's performance when given the full api @xcite and set of all corresponding modules. we refer to this as the "task-agnostic" setting ( since vipergpt relies on blip-2 as one of its modules (i.e. in simple_query for simple visual queries), we wonder how much influence blip-2 has in the vipergpt framework. moreover, how much does vipergpt gain from having modules and functions in addition to blip-2? accordingly, we run two ablations: we evaluate the performance of vipergpt without blip-2 and with only blip-2 (i.e. with no other modules). we also report these evaluations in to do so, we modify the full api prompt provided to vipergpt. for "only blip-2", we delete all modules and functions in the prompt besides imagepatch.simple_query. as the prompt for this module included in-context demonstrations relying on other (now removed) modules, we had to re-write these demonstrations. we either rewrite the existing problem ("zero-shot") or rewrite three random training set examples for each dataset ("fewshot"). for "without blip-2", we simply delete imagepatch.simple_query and all references to it from the prompt. we show examples for both procedures in appendix c. because vipergpt has no other image-to-text modules, we expect that excluding blip-2 (i.e. "without blip-2") should have a highly detrimental effect on the vqa performance of vipergpt. however, we instead observe that the variant retains 84% and 87% of the average performance, respectively, for the direct answer and multiple choice benchmarks. this indicates that including many modules improves the robustness of the vipergpt model, in that vipergpt is able to compensate by using other modules to replace blip-2. we find that using viper with blip-2 as the only module (i.e. "only blip-2") also retains significant performance in the direct answer setting: i.e. by 95% on average. moreover, this variant actually gains performance (+6% on a-okvqa and +12% on scienceqa) in the multiple choice setting. this ## decomposing problems with programs or prompting? one of vipergpt's major contributions is that it decomposes problems into python programs: it inherently gains the compositionality and logical reasoning that is built into programming languages. however, recent work in nlp suggests that questions can also be iteratively decomposed and solved more effectively than end-to-end approaches using step-by-step natural langauge prompting @xcite . here, we measure the gains related to vipergpt's choice of building logical, executable programs in python, rather than by using the interface of natural language and reasoning implicitly within llms. we want to enable as direct a comparison as possible between natural language prompting with our method and program generation with viper. thus, we choose the same vlm (blip-2) as vipergpt and an analogous llm to codex (code-davinci-002)-specifically, we use instructgpt (text-davinci-002). we present the results of our method ("successive prompting") in our prompting method performs comparably (i.e. retaining 92% of the performance on average) to vipergpt on gqa, ok-vqa, and a-okvqa in the direct answer setting, and is noticably better (i.e. +4% and +17%) on the multiple choice setting for a-okvqa and scienceqa. to the best of our knowledge, our method actually presents the highest a-okvqa multiple-choice score compared to any other training-free method. our method presents intermediate results that are in the form of natural language expressions and strictly subject to downstream operations by neural networks. on the other hand, vipergpt can present pythonic data types, like lists and numbers, as well as image regions. unlike our prompting method, vipergpt does result in a more diverse set of intermediate representations, some of which can be symbolically manipulated, and is designed to leverage a diverse set of neural networks. but from this experiment, we determine that it is not strictly necessary to decompose problems using programs in order to realize performance gains. instead, natural language prompting can offer a simpler alternative. while vipergpt leverages the intrinsic compositonality and logical execution of programming languages, our method uses conditional generation on intermediate results and a flexible natural language interface for reasoning, while remaining similarly effective. in appendix g, we tried to identify patterns in questions to determine whether they were more suitable for formal or natural language-based decomposition. we could not find any clear patterns, following simple question type breakdowns of the original datasets, but are hopeful that future work will explore this further and reveal better insights. as we have observed in sec. 4, vipergpt has been designed around a specific set of tasks (including gqa and ok-vqa), especially in its selection of modules and prompt. on the other hand, a core motivation for modular and neuro-symbolic approaches is that these should have better generalization capabilities to unseen tasks. so, we further wonder how robust vipergpt is to out-ofdistribution tasks. in particular, we consider a-okvqa and (especially) scienceqa as out-ofdistribution (compared to gqa and ok-vqa). first, we investigate changes to the prompt of vipergpt. will adding task-specific in-context examples improve the model's robustness to new tasks? in we also look at the correctness of the programs generated by vipergpt in end-to-end models. end-to-end models are the predominant approach in deep learning. in particular, we consider vision-language models here. we observe that recent, state-of-art vision-language models may rely on large-scale pretraining @xcite @xcite @xcite , be trained on vqav2 gqa ok-vqa a-okvqa q: has the food this woman is preparing been fried? q: what's around the window? q: which part of this animal would be in use if it was playing the game that would be played with the items the man is holding? q: the company that produced the device in her hand is from what country? vipergpt (only blip-2, few-shot) return simple_query ( "has the food this woman is preparing been fried?" ) @xmath0: @xmath1: @xmath2: @xmath3: "curtains" prediction: "no" artifacts: none prediction: "tail" prediction: "japan" q: what's in the image? a: a person is preparing a salad on a counter q: has the lettuce been fried? a: no prediction: "no" q: what's in the image? a: a small bedroom q: what is around the window in the small bedroom? a: curtains prediction: "curtains" q: what's in the image? a: a man carrying a dog q: what is the man doing with the dog? a: playing frisbee pred: "the dog's mouth" q: what's in the image? a: a woman sitting on a couch playing a video game q: what company produces the device in her hand? a: nintendo prediction: "japan" figure 2 : examples of decompositions for our "vipergpt (only blip, few-shot)" and "successive prompting" models on our direct answer benchmarks. these examples have been condensed for readability. in successive prompting, follow-up questions (q) are proposed by the llm (instructgpt) and answered (a) by blip-2. after a variable number of follow-ups, the llm will provide a prediction to the original question and terminate. neuro-symbolic methods. several prior methods have attempted neuro-symbolic approaches for visual reasoning tasks @xcite @xcite . however, until now, these methods were learned by training and found middling success in doing so. more recently, a few training-free approaches have been suggested that leverage the powerful in-context and program generation abilities of modern large language models @xcite @xcite . simultaneously, these approaches adopt sota neural networks in their set of modules. all together, these methods present the first competitive neurosymbolic solutions for visual reasoning tasks. natural language reasoning and tool-use. recently, it has been found in nlp that large language models can more effectively perform reasoning tasks by reasoning step-by-step @xcite . a few works have extended a similar capability in llms for complex problem and question decomposition @xcite @xcite . finally, recent works have learned ways to prompt language models to call tools @xcite . all of these emergent research directions jointly enable the prompting approach we present in sec. 5. ## conclusion in this paper, we have analyzed vipergpt @xcite , a recent and intricate modular approach for vision-language tasks. we unbox vipergpt, asking the research question: where does its performance come from? observing that one of vipergpt's five modules (i.e. blip-2 @xcite ) often outperforms or constitutes a majority of its own performance, we investigate this module's role further. through our experiments, we find that while vipergpt's marginal gains over blip-2 are a direct result of its task-specific module selection, its modularity improves its overall robustness and it can perform well even without the (seemingly critical) blip-2 module. additionally, we investigate vipergpt's choice of generating python programs. we ask if this is necessary and, alternatively, propose a method relying on prompting large language models and vision-language models to instead decompose visual questions with natural language. our method performs comparably to the relevant vipergpt variants and, to the best of our knowledge, even reports the highest multiple choice accuracy on a-okvqa @xcite by any training-free method to date.