PAELLA : Parameter-Efficient Lightweight Language-Agnostic Captioning Model (2024)

Abstract: We introduce PAELLA, a Parameter-Efficient Lightweight Language-Agnostic image captioning model designed to be both parameter and data-efficient using retrieval augmentation. The model is trained by learning a small mapping network with 34M parameters between a pre-trained visual model and a multilingual language model that is conditioned on two types of input: (i) the image itself, and (ii) a set of retrieved captions in the target language. The retrieved examples play a key role in guiding the model to generate captions across languages. Through retrieval, the model can be lightweight in terms of the number of trainable parameters, which only exist in its mapping network, and also in the amount of multilingual training data that is required. Experiments on the XM3600 dataset, featuring 36 languages, show that PAELLA can outperform or compete against some models with 3–77 \times more learned parameters and 35–863 \times more data, particularly in low-resource languages. We also find that PAELLA can be trained on only monolingual data and still show strong zero-shot abilities in other languages.

## introduction we tackle the problem of multilingual image captioning, aiming to provide textual descriptions of visual contents that can serve speakers of different languages, in contrast to most captioning models that only generate english captions. while significant progress has been made in recent years, training image captioning models has become more expensive due to the trend of scaling both data and model size @xcite . this trend is even more prominent in multilingual approaches @xcite , given the need for training data covering each target language, and the need of even larger models to mitigate the curse of multilinguality @xcite . some recent research has focused on minimizing the cost of multilingual training, such as pali-3 @xcite with 5b trainable parameters, and mblip @xcite with only 124m trainable parameters. both these approaches use pre-trained multimodal language models or pretrained visual encoders that are kept frozen, reducing the number of trainable parameters. nevertheless, both of these models still rely on training with millions or billions of examples, including in the context of image captioning alone. this paper describes a parameter-efficient lightweight language-agnostic captioning model (paella). the model is designed to be efficient, not only in terms of the number of trainable parameters, but also lightweight in the amount of multilingual training data required. paella has only 34 million trained parameters, and the model can be trained using just 566k examples, i.e., the size of the english coco dataset. paella is based on frozen pre-trained models that are augmented with retrieved examples. the only learned parameters are in a compact mapping network of cross-attention layers between a frozen clip image encoder and a frozen xglm multilingual language model. the model is trained to generate captions in the desired language using a prompt in that language. furthermore, the retrieved examples assist the model in generating meaningful captions, by providing examples of what the predicted caption should resemble. the use of retrieved examples positively contributes to reducing both the number of trainable parameters, and the required amount of multilingual data. we conduct experiments on xm3600 @xcite , an established multilingual captioning benchmark that covers geographically diverse images with human-annotated captions in 36 languages. experiments show that paella can out-perform or compete with models that are more demanding in terms of trained parameters or training data. the performance of our model in lowresource languages is particularly noteworthy, in contrast to concurrent models like mblip, that often excel in english and related languages but struggle to generalize effectively to underrepresented languages. results also show that paella demonstrates zero-shot multilingual capabilities when trained only with monolingual data such as the english coco dataset. paella achieves language transfer through retrieval, solemnly by retrieving captions in the target language during inference. ablation studies further demonstrate the benefit of our retrieval-augmented approach. ## proposed approach the parameter-efficient lightweight language-agnostic (paella) captioning model uses retrieval augmentation to generate captions in multi- image encoder retriever imágenes similares muestran || (retrieved captions in spanish) un título que puedo generar para describir esta imagen en español : immagini simili mostrano || (retrieved captions in spanish) un título que puedo generar para describir esta imagen en español similar images show a big metal pan with … a big pretty dish … a dish of assorted … a caption i can generate to describe this image in english is: multilingual lm mapping network acercamiento de una paella con camarones dentro de una olla. piatto di paella con decorazione di gamberi e cozze. macro shot of seafood paella on a plate. 🧊 🧊 🧊 🔥 en it es en es it una gran cacerola de … un gran plato … en caption 3 … una grande padella … un grande piatto … it caption 3 … a big metal pan with … a big pretty dish … a dish of assorted … … en it es top-k ple languages. an overview of the model architecture can be seen in figure 1 . we follow a similar design to the monolingual smallcap model @xcite , by building on top of powerful pre-trained unimodal models. we also use clip @xcite as the visual encoder, but instead of gpt-2 or opt as the decoder, we use a multilingual auto-regressive language model, i.e. xglm @xcite . both the encoder and the decoder are kept frozen during training, except for a newly added mapping network of cross-attention layers, that allows the decoder to attend to the visual inputs. paella generates captions conditioned on the image and on a set of k retrieved captions 2 from similar images. the retrieved captions are used to prompt the model to generate in the desired target language. the prompt follows a fixed-template which first includes examples of the k retrieved captions and ends with an instruction for the multilingual decoder to generate a caption in a desired language. the english prompt is: a caption i can generate to describe this image in [language] is: ... the prompt and captions can be tailored to different languages, by having both these parts in the desired language (see some examples of the prompts for other languages in appendix a). the parameters in the mapping network θ m are trained by minimizing the sum of the negative loglikelihood of predicting the ground truth image 2 see section 4 for details on the retrieval system. caption for each token in the sequence y 1 . . . y m , conditioned on the image v and the retrievalaugmented prompt l: we quantitatively show in section 5 that our retrieval-augmented approach has these properties: parameter-efficiency: only the cross-attention layers between a frozen encoder and a frozen decoder need to be trained. to compensate for the small number of trainable parameters, the model is guided with examples of retrieved captions. data-efficiency: through retrieval, the model does not need a huge amount of multilingual data for training, since it benefits from retrieved examples that demonstrate how to generate in the target language. we thus alleviate the data hunger of existing multilingual models, that are often trained with the same image associated to captions in multiple languages, having to repeatedly translate entire english captioning datasets for each language (e.g., coco to coco-35l @xcite ). zero-shot multilinguality: our model demonstrates multilingual capabilities even when trained only on monolingual image captioning data. it can be trained on the specific in-domain distribution from the available data in a high-resource language, and still generate in different languages. this by relying exclusively, at inference time, on retrieval augmentation in the target language from an available multilingual captioning dataset. ## results we first compare paella against state-of-theart models. we then discuss the performance of our other two variants trained on a smaller set of languages, i.e., paella core and paella mono . ## discussion we discuss paella's performance across languages in relation to the different writing systems. we then conduct ablations studies, first discussing the monolingual data required to train paella mono , followed by the importance of the retrieved information. these ablation studies were performed on the validation split of coco-35l because xm3600 only contains evaluation data. ## conclusions and future work we proposed paella, an efficient multilingual captioning model with retrieval-augmentation. contrary to previous studies, paella is lightweight to train, both in the number of parameters and multilingual data demands. results demonstrate competitiveness across languages, including low-resource languages. paella also exhibits strong zero-shot multilingual capabilities. in the future, we plan to further investigate cross-lingual transfer with monolingual supervision.