Plug-and-Play VQA : Zero-shot VQA by Conjoining Large Pretrained Models with Zero Training (2022)

Abstract: Visual question answering (VQA) is a hallmark of vision and language reasoningand a challenging task under the zero-shot setting.We propose Plug-and-Play VQA (PNP-VQA),a modular framework for zero-shot VQA.In contrast to most existing works, which require substantial adaptation of pretrained language models (PLMs) for the vision modality,PNP-VQA requires no additional training of the PLMs.Instead, we propose to use natural language and network interpretation as an intermediate representation that glues pretrained models together. We first generate question-guided informative image captions,and pass the captions to a PLM as context for question answering.Surpassing end-to-end trained baselines, PNP-VQA achieves state-of-the-art results on zero-shot VQAv2 and GQA. With 11B parameters, it outperforms the 80B-parameter Flamingo model by 8.5% on VQAv2. With 738M PLM parameters, PNP-VQA achieves an improvement of 9.1% on GQA over FewVLM with 740M PLM parameters.

## introduction recent years have witnessed unprecedented performance gains on many natural language reasoning tasks, especially in zero-shot and few-shot settings, being derived from scaling up pretrained language models (plms) and their training data @xcite @xcite @xcite @xcite . inspired by their success, a natural thought is that utilizing plms should also boost zero-shot performance in visionlanguage reasoning tasks. however, to leverage plms for vision-language tasks, most existing methods require non-trivial adaptation of the plms for the vision modality, which necessitates the design of new network components and training objectives. for example, @xcite and @xcite insert into the plms new layers that are trained from scratch. @xcite train vision encoders that output soft prompts to frozen plms. @xcite and @xcite train both the vision encoders and new layers inserted into plms. in the zero-shot setting, various vision-language pretraining objectives are employed, such as image captioning @xcite and imageconditioned masked language modeling @xcite . from the perspective of general-purpose ai, the ability to perform new tasks by simply recombining large-scale pretrained models, or foundation models @xcite , without architectural changes or extra training would be highly desirable. such a system would be able to dynamically adjust to previously unknown tasks by simply rewiring a small number of foundation models. however, to obtain high performance without some form of end-to-end training would seem difficult, if not impossible. we present plug-and-play vqa (pnp-vqa), a framework for zero-shot visual question answering which conjoins large pretrained models with zero additional training and achieves state-of-the-art performance on zero-shot vqav2 @xcite and gqa @xcite . for the purpose of bridging the vision and language modalities, we employ a pretrained vision-language model (pvlm) @xcite ) that describes visual information with textual captions. in order to obtain relevant and informative captions, we apply a network interpretability technique @xcite to detect image patches that are relevant to the question. after that, we generate captions stochastically for these image patches. finally, we employ a plm @xcite to answer the question from the captions. research in cognitive science and neuroscience suggests that the human cognitive system is largely modular @xcite . for instance, the pioneering work of fodor (1983) argued that the low-level human cognition is constituted of several fast, autonomous, and domain-specific modules. for purely practical purposes, a modular design of artificial general intelligence would make it easy to harness rapid progress in each individual component, as the components can be individually replaced and updated without affecting other parts of the system. with this paper, we offer such a modular design for zero-shot vqa that leverages recent advances in plm and pvlms and combines them with an innovative application of network interpretability. we summarize our contributions as follows: ## related work large-scale image-text pretraining of neural networks is a popular research direction. various vision-language pretraining tasks have been proposed, including image-conditioned language modeling @xcite , masked language modeling @xcite @xcite , prefix language modeling @xcite , image-text matching @xcite @xcite and image-text contrastive learning @xcite @xcite . after pretraining, several models exhibit zero-shot capabilities in image-text retrieval @xcite @xcite and image captioning @xcite . however, zero-shot vqa remains a challenging task due to its high requirement on the model's reasoning ability. adapting plms for zero-shot vqa has shown promising results. in order to incorporate vision information into plms, most existing methods perform additional vision-language training on image-text data. frozen @xcite trains the vision encoder while keeping the gigantic plm frozen to retain its knowledge in question answering. the output from the vision encoder is prepended to the text as prompts to the frozen language model. fewvlm @xcite finetunes the plm using the prefix language modeling and masked language modeling objectives. vlkd @xcite distills multimodal knowledge to plm by using clip @xcite as the teacher model during finetuning. flamingo @xcite adds additional layers to both the pretrained vision model and the plm and trains the new layers on billions of image-text pairs. different from the above work, pnp-vqa directly employs pretrained models with neither architectural modifications nor additional training. most similar to our work, pica @xcite converts an image to a single caption and adopts gpt-3 @xcite for zero-shot vqa. in comparison, pnp-vqa generates multiple question-guided captions and performs fusion of captions after encoding to effectively utilize a large number of captions, yielding considerable performance gains. an orthogonal research direction for zero-shot vqa is to train the vlms on synthetic vqa examples generated from captions @xcite . pnp-vqa does not require additional training. natural language as an intermediate representation or interface between different models or multiple steps of reasoning is an emerging machine learning strategy. it dates back to at least @xcite and saw renewed interest in the past few months due to the prevalence of large plms. @xcite and vong and lake (2022) learn natural language descriptions that function as few-shot classifiers within an image-text matching model. @xcite generate intermediate reasoning steps with finetuned plms. zhou et al. (2022) prompt a plm to generate subproblem descriptions for a complex problem, and feed the subproblems back to the plm to solve hierarchically. sampling k' patches stochastic decoding gradcam "what are the black objects?" "olives" image-question matching module image captioning module question answering module image question answer × n interpretation "salad with olives and a dressing in a bowl." caption 1: "black olives and a salad in containers." caption 2: "a salad bowl with olives." caption n: figure 1 : the system architecture of pnp-vqa, consisting of three pretrained modules: (1) an image-question matching module that identifies image patches relevant to the question, (2) an image captioning module that generates a diverse set of captions, (3) a question answering module that generates an answer given the question and captions. for the image-question matching module and image captioning module, we adopt blip @xcite . for the question answering module, we adopt unifiedqav2 @xcite . @xcite chain plm outputs and inputs. @xcite show that language-conjoined lm and vlm successfully perform captioning and retrieval but do not evaluate their models on vqa. in comparison, pnp-vqa adopts both natural language and network interpretation as the interface between different pretrained models. ## method the central idea of plug-and-play vqa (pnp-vqa) is to establish an interface between a pretrained language model and a pretrained visionlanguage model without training. we demonstrate that natural language image captions and network saliency maps together serve as an effective interface. ideally, the generated captions should thoroughly cover information that is present in the image and be relevant to the question. we foster relevance by identifying image patches most related to the question with a saliency map-based interpretability technique and generating captions from these patches only. further, we promote coverage by injecting stochasticity, including random sampling of relevant image patches and of the textual tokens during caption generation. the overall system architecture (figure 1 ) consists of three modules: in this section, we introduce the three modules in detail. ## conclusion we propose pnp-vqa, a framework with zero additional training for zero-shot vqa by conjoining off-the-shelf pretrained models. pnp-vqa leverages an image-question matching module to determine image patches relevant to the current question. an image captioning module then generates question-guided captions, which are processed by a question answering module to produce an answer. pnp-vqa achieves state-of-the-arts performance on multiple vqa benchmarks. we hope that our work will bring inspiration for further research in flexible, modular ai systems for solving vision-language tasks.