LMC ap: Few-shot Multilingual Image Captioning by Retrieval Augmented Language Model Prompting (2023)

Abstract: Multilingual image captioning has recently been tackled by training with large-scale machine translated data, which is an expensive, noisy, and time-consuming process. Without requiring any multilingual caption data, we propose LMCap, an image-blind few-shot multilingual captioning model that works by prompting a language model with retrieved captions. Specifically, instead of following the standard encoder-decoder paradigm, given an image, LMCap first retrieves the captions of similar images using a multilingual CLIP encoder. These captions are then combined into a prompt for an XGLM decoder, in order to generate captions in the desired language. In other words, the generation model does not directly process the image, instead it processes retrieved captions. Experiments on the XM3600 dataset of geographically diverse images show that our model is competitive with fully-supervised multilingual captioning models, without requiring any supervised training on any captioning data.

## introduction the task of image captioning has witnessed impressive performance gains with the trend of large-scale encoder-decoder models and vision-and-language pre-training @xcite @xcite . despite all of this progress, existing models are mostly available on english or are specialised for other high-resource languages. this limits the access to the technology for a broader range of languages that exist in the world. moreover, the current mainstream trend results in design decisions and methods that may only work well for english-centric datasets or the couple of languages for which captioning data is available @xcite . there is a need to develop multilingual image captioning models that can serve speakers of different languages. still, scaling captioning models to a wide variety of languages involves different challenges. one major limitation is the lack of multilingual imagecaption pairs of clean labelled data for training the models. one possible solution is to automatically translate the existing english datasets @xcite . while effective, this approach can result in models that learn translation artefacts, and perpetuates an english-centric perspective instead of encouraging the use of geographically diverse concepts that are not overly specific to the western culture @xcite . moreover, with or without automatic translations, training captioning models with multilingual data can be expensive, given the amount of data and number of parameters needed to mitigate the curse of multilinguality @xcite . this paper presents lmcap, an image-blind multilingual image captioning model that does not require any training specific for image captioning. we propose an efficient method that reuses a pretrained multilingual language model and adapts it to the vision-and-language captioning setting. our work is motivated by the recent "socratic models" framework @xcite , in which different models can be combined through text prompting (e.g., image captioning can be achieved by prompting a language model with a set of visual concepts extracted from the predictions of a vision model). different from the original socratic models, our approach is inspired by retrieval-augmented generation @xcite . specifically, a multilingual language model generates captions given a prompt consisting of the captions retrieved from similar images, and a demonstration of how to produce a caption in the desired language. we note here that this is an image-blind approach, i.e. the language model producing the caption does not actually process the image. our main contributions are as follows: (1) we propose a few-shot multilingual image captioning approach named lmcap, that re-uses pre-trained models without requiring any training specific for image captioning; (2) to the best of our knowledge, lmcap is the first captioning model with retrieval-augmented generation in a multilingual setting, and in a few-shot setting of captioning; (3) we report on experiments with the xm3600 benchmark @xcite of human-authored captions and geographic diverse images, demonstrating that lmcap exhibits strong few-shot performance on a wide variety of languages; (4) we further show that lmcap performs substantially better than the original socratic models. moreover, instead of only achieving competitive performance against other zero-shot models, lmcap can also compete with a large-scale supervised state-of-art captioning model. ## background and related work image captioning: the task of automatically generating textual descriptions for input images has been largely explored in english, while multilingual image captioning has only been addressed in a couple of studies @xcite @xcite . like in most recent work on image captioning @xcite @xcite , studies addressing multilingual setups have also focused on scaling the size of encoder-decoder models and the amount of training data, resorting to machine translated versions of multimodal data to accommodate multiple languages @xcite . differently from training a large-scale encoder-decoder model, we follow a few-shot setting with an image-blind approach based on prompting. few-shot and zero-shot approaches: performing few-shot learning by prompting a language model with examples and demonstrations of a task @xcite @xcite ) is an efficient and effective alternative to update model parameters. similarly to other nlp tasks, recent work in the vision-andlanguage domain has used prompt-based learning by building on top of pre-trained language and vision models, although usually also involving extra multimodal training @xcite @xcite . in our work, we follow a similar few-shot prompting approach to the recent socratic models @xcite that do not involve any multimodal training, as described next. in image captioning, there have also been zero-shot methods that similarly to our approach do not involve any training, by relying on prompts or adaptations over the decoding algorithms, such as zerocap @xcite and conzic @xcite . however, these models work for english and not for the multilingual captioning setting. @xcite proposed the socratic models (sms) framework, where different multimodal pre-trained models communicate via zero-shot or few-shot prompting. for the task of image captioning, sms generate captions by prompting a language model (i.e., gpt-3 @xcite ) with information about the input image obtained with another pre-trained model (i.e., clip @xcite ). the visual information is in this way represented into a languagebased prompt, containing the number of people presented in the image, the places, objects, and what is the type of image. we explore a similar approach in the multilingual setting by reusing multilingual models, and through a retrieval-based prompt. ## model language model prompt-based captioning (lmcap) is a few-shot multilingual captioning model augmented with retrieval. it involves prompting a language model (lm) with captions retrieved from a datastore by a vision-and-language model (vlm). captions are generated in an image-blind manner, without actually processing the visual contents of the input image, instead using a prompt containing the retrieved captions. the method works as follows: first, given an input image, the vlm is used to find relevant captions in the datastore. second, the retrieved captions are converted to a language prompt, which is encoded by the multilingual lm to generate captions in a desired language, conditioning the generation on the prompt. finally, the set of generated captions can be scored by the vlm against the input image, to select the best caption. the main aspects of our approach are shown in figure 1 and fully detailed next. image-text retrieval: the input image and a datastore of captions are encoded by a multilingual clip @xcite , i.e. a vlm that can be used to calculate image-text similarity. in this way, given the encoded data, m-clip is used to retrieve the k most similar captions from the datastore. the datastore contains captions associated to diverse images, which can be in english or another language. the retrieved captions will serve to guide a language model as an example of what the predicted caption should resemble, through the use of a prompt and as described next. retrieval-augmented prompting: the retrieved captions, which represent the visual information about the image, are formatted into a prompt for the language model. the prompt starts with fixed n -shot examples and ends with the retrieval information about the input image, to guide the language model. each shot is a demonstration of how to generate a caption in a desired language for an image, given a set of retrieved captions. after these nexamples, the prompt terminates with the retrieved information about the actual input image. an example of the format of the prompt can be seen in figure 1 and in more detail in appendix d. we note that the retrieved captions, either from the fixed n -shot examples or those corresponding to the input image, can be presented in any language or in multiple languages. prompting multilingual text generation: the aforementioned prompt is used as input for an xglm @xcite pre-trained multilingual autoregressive lm, to generate captions in a given language. xglm is applied in a few-shot setting, which means that lmcap does not require any training (i.e., the captions are generated by providing the prompt at inference time to xglm). captions are generated in the desired language by including an example in the n demonstrations in the prompt, as shown in figure 1 . multilingual reranking: after the lm generates a set of captions, the multilingual vlm performs a final image-text similarity step to find the caption that best describes the input image. this is based on the same m-clip model used for the initial image-text retrieval. ## evaluation in this section, we describe the evaluation of lm-cap. we describe the experimental setup and results, and we also present ablation studies and further discussions about our approach. ## conclusions this paper proposes lmcap, an image-blind fewshot multilingual image captioning model. lm-cap is based on prompting a language model with n -shot examples and retrieved captions extracted by a vision-and-language model, to condition caption generation in a desired language with a multilingual language model. on xm3600, i.e. a humanlabelled massively multilingual multimodal benchmark, lmcap performs competitively against the state-of-the-art without involving expensive training with large-scale translated multimodal data, or with any captioning data. experimental results further demonstrate that lmcap largely outperforms socratic models @xcite , showing that retrieval augmentation plays a crucial role in our prompting approach. as future work, we plan to further assess the use of multilingual data in the datastore, as well as the impact of directly promoting diversity @xcite in the captions used in the prompt.