Zero-shot Visual Question Answering with Language Model Feedback (2023)

Abstract: In this paper, we propose a novel language model guided captioning approach, LAMOC, for knowledge-based visual question answering (VQA). Our approach employs the generated captions by a captioning model as the context of an answer prediction model, which is a Pre-Trained Language model (PLM). As the major contribution, we leverage the guidance and feedback of the prediction model to improve the capability of the captioning model. In this way, the captioning model can become aware of the task goal and information need from the PLM. To develop our approach, we design two specific training stages, where the first stage adapts the captioning model to the prediction model (selecting more suitable caption propositions for training) and the second stage tunes the captioning model according to the task goal (learning from feedback of the PLM). Extensive experiments demonstrate the effectiveness of the proposed approach on the knowledge-based VQA task. Specifically, on the challenging A-OKVQA dataset, LAMOC outperforms several competitive zero-shot methods and even achieves comparable results to a fine-tuned VLP model. Our code is publicly available at https://github.com/RUCAIBox/LAMOC .

## introduction recently, pre-trained language models (plms) @xcite , especially large language models @xcite have demonstrated excellent capabilities in solving tasks that require background knowledge or complex reasoning, such as commonsense reasoning @xcite and logical reasoning @xcite . inspired by these successes, recent studies have proposed utilizing plms 1 to solve complex vision-language corresponding author. 1 in this paper, plms refer to the models trained on textonly corpus, instead of the text encoder/decoder in visionlanguage pre-trained (vlp) models, which typically have a weaker reasoning capacity in linguistic content. ## method in this section, we present the proposed lamoc: language model guided captioning method for vqa. the overall architecture of lamoc is depicted in figure 2 . ## experiment this section shows the experimental setup and then highlights the main conclusions of our results. ## conclusion in this paper, we propose lamoc, a language model guided captioning method that improves a captioning model to generate comprehensive captions for an image to help answer the question. in order to train such a model, we first perform captioning adaptation on a self-generated dataset filtered by flan-t5-xxl, and then fine-tune the updated captioning model through reinforcement learning from plm feedback. our method, lamoc, generates captions that are both informative and able to assist plms in vqa tasks, as demonstrated through experiments on two knowledge-based vqa datasets. on the challenging a-okvqa dataset, lamoc substantially outperforms previous zero-shot methods and even achieves comparable results to a fine-tuned vlp model. additionally, we show that lamoc is generalizable to plms of varying sizes, from 223m to 11b parameters, demonstrating its potential to be applied to llms, which we leave as future work.