Modular Visual Question Answering via Code Generation (2023)

Abstract: We present a framework that formulates visual question answering as modular code generation. In contrast to prior work on modular approaches to VQA, our approach requires no additional training and relies on pre-trained language models (LMs), visual models pre-trained on image-caption pairs, and fifty VQA examples used for in-context learning. The generated Python programs invoke and compose the outputs of the visual models using arithmetic and conditional logic. Our approach improves accuracy on the COVR dataset by at least 3% and on the GQA dataset by 2% compared to the few-shot baseline that does not employ code generation.

## introduction the scope of reasoning needed for visual question answering (vqa) is vast, and demands the synthesis of many skills -from grounding language to pixels @xcite @xcite and spatial reasoning @xcite to commonsense and knowledgebased reasoning @xcite . consider the question "is the carriage to the right of a horse?". to consistently answer such questions correctly, a system must recognize that the question is the conjunction of two subquestions: "is there a horse?" and "is the carriage to the right of the horse?" scaling the typical finetuning paradigm to all possible combinations of reasoning skills is prohibitively expensive in annotation cost and makes it difficult to add skills to an already-trained system. modular approaches, on the other hand -from classic methods @xcite , to differentiable neural module networks (nmns) @xcite @xcite ) -offer a potential route to leverage and scale to the compositional nature of visual reasoning as a means to generalize: i.e., infinite use of finite means. however, the modules of an nmn must still be trained jointly on a large dataset, and are also restricted in that they (i) require a parser, which must be modified if modules are added or removed from the system, and (ii) require retraining if a module is replaced. in this work, we investigate an alternative class of modular vqa approaches, whereby building on the recent advent of highly capable out-of-the-box language models (lms) @xcite and visual language models (vlms) @xcite , we develop systems that formulate vqa as a program synthesis problem. specifically, our method codevqa, illustrated in figure 1 , uses code-writing lms to take questions as input, and outputs code to (i) orchestrate a series of visual primitive apis that wrap around vlms to probe the image for specific pieces of visual information (e.g., captions, pixel locations of entities, or image-text similarity scores), and (ii) reason about that information with the full expression of python code (e.g. arithmetic, logic structures, feedback loops, etc.) to arrive at an answer. from a practical perspective, the modularity of codevqa combined with the few-shot prompting capabilities of lms enable it to adapt to a broad range of desired vqa label distributions without additional model training, and benefits from replacing individual modules with improved versions as they become available. we evaluate codevqa in the few-shot vqa setting, which has seen a great deal of recent work @xcite @xcite . our method outperforms previous approaches by at least 3% on the covr dataset @xcite , which requires reasoning over multiple images, and by 2% on the gqa dataset @xcite . our results suggest that the benefits of modularity with recent off-the-shelf models can be realized in vqa without additional model training. 1 code generation @xmath0, "is there a horse?") @xmath1: carriage_pos_x,@xmath2,"carriage") horse_pos_x, @xmath3, "horse") if @xmath4: @xmath5 is the carriage to the right of a horse? answer: no codex (few-shot prompting) in-context examples # image 1: on which side of the picture is the rug? @xmath6.jpg") rug_pos_x, @xmath7, "rug") if @xmath8: @xmath9: @xmath10. . . query(img,"is there a horse?") execute code get_pos(img,"carriage") get_pos(img,"horse") @xmath11, 11 returns 12, 11 returns "yes" captions: several recent approaches for reasoning tasks consist of an lm that writes programs and an interpreter for these programs. @xcite applies this approach to robotics. @xcite introduces a framework for reasoning jointly over tables, text, and images, where the images are represented by image captions. @xcite used a syntactic parser and hard-coded rules rather than an lm to aggregate outputs from clip @xcite for zero-shot referring expression comprehension; their finding that clip is not useful for spatial keywords motivates our code generation approach to spatial reasoning. concurrent with our work, other papers have introduced similar frameworks for multi-hop vqa @xcite . these papers conflate the benefit of program synthesis with the benefits of the lm, in-context examples, and vision models used as primitives. by contrast, we analyze the effect of program synthesis by comparing codevqa against a strong lm-based few-shot baseline using the same in-context example selection method. moreover, while these frameworks rely on supervised vqa or object detection models, we show that we can obtain comparable performance (on the gqa dataset) using only the lm and models pre-trained on image-text pairs. ## few-shot vqa via code generation in visual question answering (vqa), the inputs to the system are an image and a question and the output is a textual answer. we consider the fewshot vqa setting in which the system has access to only a small number (50) of human-annotated vqa instances. overview. fig 1 illustrates our approach. given an image and a corresponding question, codevqa first generates a python program using just the question. it then executes this program, using the image when necessary, to predict the answer. we first define the set of code primitives that our system uses ( § 3.1). then we describe how we generate a program that composes these primitives based on the question ( § 3.2). finally, we enumerate the pre-trained models that we employ ( § 3.3). ## conclusion in this paper, we have introduced a framework for modular few-shot vqa. our approach prompts an lm to generate a python program that invokes pretrained visual modules and composes the outputs of these modules to predict the answer. unlike previous modular vqa techniques, this framework does not require (re-)training modules or a parser. also, obtaining interpretable module outputs from previous modular approaches is nontrivial @xcite , whereas in our approach the modules are frozen and thus interpretable. codevqa can also be viewed as expanding pipelined systems @xcite to the full expression of code. our ap-