Modularized Zero-shot VQA with Pre-trained Models (2023)

Abstract: Large-scale pre-trained models (PTMs) show great zero-shot capabilities. In this paper, we study how to leverage them for zero-shot visual question answering (VQA).Our approach is motivated by a few observations. First, VQA questions often require multiple steps of reasoning, which is still a capability that most PTMs lack. Second, different steps in VQA reasoning chains require different skills such as object detection and relational reasoning, but a single PTM may not possess all these skills. Third, recent work on zero-shot VQA does not explicitly consider multi-step reasoning chains, which makes them less interpretable compared with a decomposition-based approach. We propose a modularized zero-shot network that explicitly decomposes questions into sub reasoning steps and is highly interpretable. We convert sub reasoning tasks to acceptable objectives of PTMs and assign tasks to proper PTMs without any adaptation. Our experiments on two VQA benchmarks under the zero-shot setting demonstrate the effectiveness of our method and better interpretability compared with several baselines.

## introduction visual question answering (vqa), the task of answering textual queries based on information contained in an image, is a multimodal task that requires comprehension and reasoning of both visual and textual content @xcite . most previous work on vqa either trains vqa models from scratch (e.g., @xcite ; @xcite ) or fine-tunes pre-trained vision-language models for vqa (e.g., @xcite ; @xcite ). thus, they rely heavily on labeled vqa data, which are expensive to obtain. vqa models based on supervised learning are also hard to generalize to new domains or new datasets @xcite @xcite . recently, large-scale pre-trained models (ptms) have demonstrated strong transferability to different downstream tasks under zero-shot settings, i.e., without any training data for the downstream tasks @xcite . with increased pre-training data size, these models show strong zero-shot performance on various down-stream tasks, such as image classification and face detection with the clip model @xcite and sentiment analysis and commonsense question answering with the gpt-3 model @xcite . however, few studies have focused on zero-shot vqa from pre-trained models. despite the power of these ptms, it is not straightforward to directly apply them to vqa under zero-shot settings, because they are not pretrained with the same objective as vqa. some recent work converts images to tokens that pretrained language models can understand so that vqa can be converted to text-based qa @xcite @xcite @xcite . however, this approach requires either a strong pre-trained image captioning model that can capture sufficient visual details or auxiliary training to obtain such a captioning model. some other work converts vqa into a multimodal matching problem so that pretrained vision-language models (pt-vlms) such as clip can be used @xcite . however, complex vqa questions such as those found in the gqa dataset @xcite often require spatial reasoning and/or multi-step reasoning, which pt-vlms may not be strong at @xcite . vqa questions can be complicated and often require different reasoning steps such as object detection and spatial reasoning, as the example question in figure 1 illustrates. previously, people proposed neural module networks @xcite , which are modularized net- question: the man holding controller is to the left or right of the woman? parser find(controller) relocate([1], holding) filter([2], man) choose([3], [4],[left, right]) find(woman) pre-trained models traditional nmns find relocate choose query …… vqa data model function owl object detection mdetr modulated detection clip mm matching mdetr(man holding controller) owl(woman) clip([1], [2], [left, right]) layout mapping the man is to the [ans] of the woman right 0.25 …… left 0.75 …… ans. voc. works where each pre-defined module performs a specific reasoning task. these pre-defined modules are trained end-to-end from labeled vqa data. motivated by the idea of modularization, in this paper, we propose a modularized zero-shot network for vqa (mod-zero-vqa) by decomposing questions into sub-tasks and assigning appropriate sub-tasks to ptms without any adaptation. given a question, we first parse the question into basic reasoning steps explicitly. these reasoning steps will then be reconfigured and mapped to different ptms based on a set of rules we define. specifically, we consider the following ptms: owl @xcite as the object detector, mdetr (kamath et al., 2021) for reference expression localization (including several skills such as relational and spatial reasoning) and clip @xcite ) as the answer generator for open-ended questions. considering the limited capabilities of current pretrained vision-language models in spatial relation understanding @xcite , we also define simple and general heuristics to aid spatial reasoning. note that only when we decompose questions and reasoning chains step by step can we insert human heuristics for spatial reasoning, because we have the intermediate outputs such as objects' bounding boxes from previous steps. we evaluate the proposed method on the gqa dataset @xcite where questions are compositional and require multi-step reasoning. the experiment result shows that the proposed model surpasses the baselines significantly on gqa, with near 13% of relative improvement over the strongest baseline (from 41.9 to 47.3). the results confirm the benefit of modualarization when using ptms for zero-shot vqa. in addition, our method is interpretable because of the explicit reasoning steps generated. the contributions of our work can be summarized as follows: (1) we propose a novel modularized zero-shot vqa method that utilizes different pre-trained models for different reasoning steps; (2) we design rules to map different vqa reasoning steps to suitable ptms so that we can leverage these ptms without any adaptation; 3) experiment results show the effectiveness of the proposed method, especially when questions consist of multiple steps of reasoning. existing zero-shot vqa methods. work on zero-shot vqa is very limited. we can organize existing work into the following categories. one line of work leverages the question answering capability in pre-trained language model (lms). some of them adopt prefix language modeling with weakly-supervised data other than vqa data (i.e., image-text pairs) to convert visual information into discrete tokens (prefix) that lms can understand. frozen @xcite , vlkd @xcite and fewvlm @xcite fall under this category. some directly convert vqa images into textual descriptions so that the task of vqa changes to text-based qa and lms can be applied. methods in this category include pica @xcite and pnp-vqa @xcite . recent work @xcite converts vqa to an image-text matching problem and prompts the clip model @xcite , a large-scale vision-language model pre-trained on the image-text matching task. the prompts can be either question irrelevant such as quesion: [ques]; answer: [mask] (qip by @xcite ) or question-related by converting questions into a masked statement (tac-p by @xcite ). however, a limitation with these methods is that several of them still require training, although the training data is not in the form of vqa. besides, converting images to captions and leveraging textbased qa may lose important visual details during the caption generation step. the two methods above using clip do not address the issue that clip model lacks compositional and spatial reasoning abilities, which has been observed in previous work @xcite . ## modularized zero-shot vqa our method is motivated by neural module network (nmn) based vqa, which decomposes questions into reasoning steps, where each module in the nmn is pre-defined to perform a specific reasoning task. the idea allows us to select appropriate pre-trained models to handle different reasoning tasks in a question. specifically, in nmn-based vqa, we first manually define a set of reasoning steps such as object detection and spatial reasoning, each represented by a module. a question is then explicitly decomposed and converted into a layout of modules, which is an executable program showing the reasoning chain to reach the final answer. the top section of figure 1 shows the layout corresponding to the sample question. to train an nmnbased vqa system, usually a layout generator is separately built first, which either uses hand-crafted rules over dependency parses of questions or is a trained seq2seq model. then, the parameters of the various vqa modules are learned from vqa training data. for our work, we do not want to use vqa data for training. but we observe that many modules in nmn-based vqa can be supported by pre-trained models that have already acquired the capabilities needed by these modules. the key component of our method is therefore to map a layout of modules produced by traditional nmn-based vqa to a simplified layout of zero-shot components that can be implemented directly using pre-trained models. ## conclusion and future work in this work, we propose a modularized zero-shot vqa method, motivated by the idea of neural module network (nmn). instead of training modules in nmn with vqa data, we decompose questions into reasoning tasks explicitly, leverage pre-trained models and assign proper reasoning tasks to them. experiments show that our model is powerful on questions requiring multi-step reasoning and applicable for real-world vqa. besides, the proposed model is highly interpretable, which helps to pinpoint weaknesses of a vqa system, making it easier to improve a system. our model highlights a future direction of leveraging pre-trained models for other complicated tasks requiring multiple reasoning capabilities.