Re- V i LM : Retrieval-Augmented Visual Language Model for Zero and Few-Shot Image Captioning (2023)

Abstract: Augmenting pretrained language models (LMs) with a vision encoder (e.g., Flamingo) has obtained state-of-the-art results in image-to-text generation. However, these models store all the knowledge within their parameters, thus often requiring enormous model parameters to model the abundant visual concepts and very rich text descriptions. Additionally, they are inefficient in incorporating new data, requiring a computational-expensive fine-tuning process. In this work, we introduce a Retrieval-augmented Visual Language Model, Re-ViLM, built upon the Flamingo, that supports retrieving the relevant knowledge from the external database for zero and in-context few-shot image-to-text generations. By storing certain knowledge explicitly in the external database, our approach reduces the number of model parameters and can easily accommodate new data during evaluation by simply updating the database. We also construct an interleaved image and text data that facilitates in-context few-shot learning capabilities.We demonstrate that Re-ViLM significantly boosts performance for image-to-text generation tasks, especially for zero-shot and few-shot generation in out-of-domain settings with 4x less parameters compared with baseline methods.

## introduction image-to-text generation, also known as image captioning (e.g., @xcite , plays a vital role in understanding visual information and enhancing human-ai interaction. this task has a wide range of practical applications such as gaming, virtual reality, and robotics @xcite @xcite . to address this *equal contribution. ‡work done during an internship at nvidia. 1 uiuc. 2 nvidia. 3 ut austin. 4 university of wisconsin-madison. 5 california institute of technology. †equal advising. correspondence to: zhuolin @xmath0.edu>, wei @xmath1.@xmath2, numerous methods have been proposed recently and obtained great success (e.g., @xcite @xcite @xcite . among them, visual language models (lms) build on top of pretrained autoregressive lms (e.g., @xcite , and inherit its powerful text generation ability. in particular, the pretrained lm parameters are usually frozen and only some trainable layers (e.g., adaptor) are added into the large lm during multimodal pretraining @xcite @xcite . this frozen lm strategy can avoid catastrophic forgetting when the visual lm is trained on ⟨image, text⟩ data, where the text quality is usually lower than the text-only corpus to pretrain lm. in addition, it enables the compelling zero-shot or few-shot capability of pretrained lm (e.g., flamingo, @xcite . however, such methods have two major limitations: 1) they store all acquired knowledge within the model parameters, making them parameter inefficient in modeling the abundant visual concepts (e.g., uncommon objects) and rich textual descriptions (e.g., alternative descriptions for the same scene). 2) they are inefficient in incorporating new data, typically requiring computationally expensive fine-tuning @xcite or pertaining on increasingly more parameters and interleaved ⟨image, text⟩ data @xcite . in the past few years, retrieval-augmented lms @xcite @xcite have shown notable success in improving accuracy while reducing model parameters by retrieving large-scale text corpus. despite their success, there are several issues to be addressed before we apply retrieval technique to visual lms for image-to-text generation: i) the visual lm needs to seamlessly retrieve and encode external knowledge at the beginning of multi-modal pretraining. otherwise, the flamingo: a brown and black animal in a pool of water. ## related work visual language models many recent work tackle the problem of generating text captions for given images (e.g., @xcite @xcite @xcite @xcite . among these work, visual language models @xcite directly augment pretrained lms with visual component, achieving strong results in both zero-shot and few-shot generation. retrieval-augmented models retrieval has been successfully applied in various nlp tasks, including question answering @xcite , autoregressive language modeling @xcite , and other knowledge-intensive tasks (e.g., @xcite . in computer vision, retrieval has also been applied for image recognition with long-tail distribution of classes @xcite . in this work, we apply retrieval for image captioning. ra-cm3 @xcite augments the cm3 backbone @xcite with retrieval, which can perform both image-to-text generation and text-to-image synthesis. in contrast to our revilm, there are the following differences: 1) we investigate the "copy-and-paste" behavior of retrieval-augmented model during training, and propose a simple filtering strategy during retrieval. in contrast, ra-cm3 proposes a query-dropout strategy that drops some tokens of the query caption used in retrieval. in our ablation study, we find our simple strategy works better than the dropout regularization as shown in § 5.5.1. 2) our re-vilm uses retrieval-augmented lm decoder layer with cross-attention module to attend to the retrieved similar captions. in contrast, ra-cm3 appends the retrieved captions as the prefix context on the decoder side. in our ablation study, we find ours are more effective, as shown in § 5.5.2. 3) for image-to-text generation, ra-cm3 only provides the result of 2-shot in-context image captioning on mscoco @xcite . in contrast, we perform extensive evaluations of our re-vilm on various benchmarks under zero-shot, few-shot and fine-tuning settings. ## re-vilm architecture in this section, we begin by outlining the framework of re-vilm. after that, we delve into the details of each component in depth. ## experiments in this section, we evaluate the performance of re-vilm under three different settings: zero-shot, fewshot and fine-tuning, on various image captioning benchmarks. we aim to demonstrate the superiority of our retrieval augmentation technique in improving the quality and relevance of generated captions through retrieving relevant knowledge from external databases. we compare our results to several widely-used image captioning models such as enc-dec @xcite , simvlm @xcite , and flamingo @xcite . through extensive evaluation of re-vilm, we conclude that re-vilm is compelling under zero-shot and few-shot settings. ## limitations in this paper, we focus on exploring emergent zero-shot and in-context few-shot image captioning. to achieve this, we designed our retrieval augmented model mainly based on the flamingo framework @xcite , and leave the application of our retrieval design to other imageto-text frameworks @xcite @xcite as future work. furthermore, since there is no official implementation of flamingo and its training datasets, our framework is based on our reimplemented flamingo, trained on publicly available datasets and manually crafted interleaved image-text datasets. also from the scaling perspective, comparable to gpt in the text-only domain, one of the most important advantages of the flamingo-like model is scaling. in this study, we haven't been able to further scale re-vilm to 80b to address the benefits from retrieval on large scale visual language models.