Retrieval-augmented Image Captioning (2023)

Abstract: Inspired by retrieval-augmented language generation and pretrained Vision and Language (V&L) encoders, we present a new approach to image captioning that generates sentences given the input image and a set of captions retrieved from a datastore, as opposed to the image alone. The encoder in our model jointly processes the image and retrieved captions using a pretrained V&L BERT, while the decoder attends to the multimodal encoder representations, benefiting from the extra textual evidence from the retrieved captions. Experimental results on the COCO dataset show that image captioning can be effectively formulated from this new perspective. Our model, named EXTRA, benefits from using captions retrieved from the training dataset, and it can also benefit from using an external dataset without the need for retraining. Ablation studies show that retrieving a sufficient number of captions (e.g., k=5) can improve captioning quality. Our work contributes towards using pretrained V&L encoders for generative tasks, instead of standard classification tasks.

## introduction image captioning is the task of automatically generating a short textual description for a given image. the standard approach involves the use of encoderdecoder neural models, combining a visual encoder with a language generation decoder (see @xcite for a survey). in early studies, the encoder was typically a convolutional neural network model (cnn) pretrained on the imagenet classification dataset @xcite or a pretrained faster-rcnn object detector @xcite , whereas the decoder was commonly an lstm @xcite together with an attention mechanism @xcite . more recently, transformer based models have been achieving state-of-the-art results on a variety of language processing @xcite @xcite and computer vision tasks @xcite . accordingly, state-of-the art image captioning models have replaced the conventional cnn-lstm approach with encoder-decoder transformers @xcite . still, in both cases, the encoder only attains visual representations, whereas richer features could be captured from image-text interactions if the encoder had access to useful textual context related to the input image (e.g., sentences associated to similar images). in this paper, we present a new type of image captioning model that uses a pretrained v&l bert @xcite @xcite to encode both the input image and captions retrieved from similar images. this model generates captions conditioned on representations that consider linguistic information beyond the image alone. moreover, specifically using the retrieved captions as textual contexts rather than other alternatives (e.g., image tags or object names) can aid guiding the language generation process, since the model is now provided with wellformed sentences that are semantically similar to what the predicted caption should resemble. in experiments on the coco dataset @xcite , the proposed model is competitive against state of the art methods. in a series of ablation experiments, we find that the model improves when encoding multiple retrieved captions, and that it could reach better performance if it was able to retrieve better captions from the datastore. in experiments on the smaller flickr30k dataset, we show that allowing the model to retrieve captions from the larger coco dataset can improve performance without needing to retrain the model. we hope that our work inspires the adoption of pretrained v&l encoders for a broader range of generative multimodal tasks. there have been several recent studies proposing v&l berts to learn generic multi-modal representations with large amounts of paired image and text data, which can then be fine-tuned to downstream tasks. however, these pretrained models have mostly been applied to classification tasks and have seen limited use for image captioning, a task which typically only considers single-input images, as opposed to image-text pairs, as proposed in this work. ## model we present a model that captions images, given both the image and a set of k captions retrieved from similar images using a retrieval system. this approach belongs to the class of retrievalaugmented language generation models @xcite . in our model, the image and the retrieved captions are jointly encoded using a pretrained v&l encoder to capture cross-modal representations in the combined input data. we denote our model as ex-tra: encoder with cross-modal representations through retrieval augmentation. it consists of three components, namely an encoder, a retrieval system, and a decoder. ## results overall, extra is competitive to state-of-the art captioning models. it outperforms captioning models with vision encoders, and vl-t5, which, like extra, uses a v&l encoder, but with object tags as linguistic inputs rather than retrieved captions. although extra does not outperform the state of the art captioning model, camel, that uses a dual decoder, it outperforms the variant of camel that uses the same faster-rcnn features. extra also competes with general purpose v&l bert models. notice that our approach can be adapted to other v&l encoders besides lxmert (e.g., os-car, vinvl, etc.), or to more powerful decoders (e.g., as in camel). likewise, other models could benefit from retrieval-augmentation with captions. ## conclusions we propose extra, a retrieval-augmented image captioning model that improves performance by exploiting cross-modal representations of the input image together with captions retrieved from a datastore. extra make uses of a pretrained v&l bert, instead of an image-only encoder, combined with a language decoder. to generate a caption, the decoder attends to the cross-modal encoder features, containing information from image regions and also textual evidence from the retrieved captions. image captioning is therefore addressed as language generation conditioned on vision and language inputs, instead of vision only. to evaluate this model, extra was assessed against strong encoder-decoder models in the area, and ablation studies were also conducted. the experiments conducted on the coco dataset confirmed the effectiveness of the proposed captioning approach. for future work, we plan to explore the utility of extra in out-of-domain and in few-shot learning settings, since the retrieval component can be easily modified to include external datastores, without the need to retrain the whole model. we also plan to explore how this approach can be adapted to other powerful vision and language encoders besides lxmert. finally, we will explore methods that allow us to jointly train the retrieval mechanism with the full model in order to retrieve captions that are more similar to the input image.