V - G lór IA - Customizing Large Vision and Language Models to E uropean P ortuguese (2024)

Abstract: Generative Vision and Language models have obtained remarkable results recently, thanks to the use of robust pre-trained Visual encoders and Large Language Models (LLMs), together with efficient model adaptation training strategies, requiring minimal architecturalmodifications, while preserving LLMs’ original capabilities. With these advances focusing mainly on the English language, there is a gap in customization methodologies for other languages. In this paper, we propose a customization methodology that adapts existingstate-of-the-art vision and language architectures to European Portuguese (PT-PT). As a result of applying this methodology, we introduce V-GlórIA , the first Large Vision and Language generative model specifically customized for European Portuguese. V-GlórIA supports multimodal tasks such as image captioning, retrieval, and dialogue. To deliver V-GlórIA, we leverage state-of-the-art V&L architectures, and contribute with PT-PT machine-translated pre-training (CC3M PT-PT) and benchmark (MSCOCO PT-PT and VisDial PT-PT) datasets.Our experiments show that V-GlórIA delivers promising performance in text-image retrieval and downstream tasks in a zero-shot setting, such as image captioning and visual dialogue tasks, highlighting the effectiveness of our customization approach.

## introduction vision and language are two of the main communication and information perception mediums, serving as fundamental channels through which humans interpret and interact with the world around them. devising vision and language (v&l) models that can seamlessly combine these two modalities is paramount to delivering ai systems capable of addressing tasks such as image captioning and visual question-answering, essential tasks to aid visually impaired individuals, and imageto-text and text-to-image retrieval, for general multimodal information seeking. recently, there have been notable advances in vision and language models @xcite @xcite , which leverage large language models as backbones @xcite @xcite . most of these advances have been made with models in english or other high-resource languages, leaving behind other lower-resource languages, as is the case of european portuguese (pt-pt). this evidences the urgent need of having effective customization methologies to deliver v&l lms, openly available, for pt-pt speakers. this customization process raises two complementary challenges: 1) how to overcome the limited availability of pt-pt multimodal datasets and resources, and 2) how to train a large visiwon and language model, capable of addressing multiple v&l tasks, in pt-pt. most llms are trained with text-only web scraped data, achieving great performance on a myriad of natural language tasks, but lack an overall understanding of images, thus not having visual reasoning capabilities. pioneering vision and language models, adopted fully multimodal transformer-based models @xcite @xcite , with either singlestream or dual-stream architectures @xcite , pre-trained on image-text pairs. more recently, towards generalizing high-performing large lms to the visual domain, it is common practice to leverage text-only llms as the backbone and equip them with a visual encoder @xcite . then, llms are augmented with a visual projection component that aligns visual tokens with the llm tokenspace @xcite . in this paper, we seek to establish a v&lcustomization methodology to european portuguese, and as a result, deliver the first european portuguese vision and language lm, v-glória. to this extent, we make two major contributions: 1) we create and make available both large-scale image-text pre-training datasets as well as wellknown v&l benchmarks in european portuguese. in particular, cc3m @xcite ) pt-pt (3 million image-caption pairs) for pre-training, mscoco @xcite pt-pt (image-caption pairs) and visdial @xcite pt-pt (visual dialogs) for benchmarking on downstream v&l tasks. an extensive assessment of available machine translation approaches is carried out. 2) following the v&l lms state-of-the-art, we adapt the fromage @xcite model to support pt-pt. its flexible decoder-based architecture, augmented with multimodal specialized layers, gives the model the capacity to process and produce interleaved multimodal inputs and outputs. given that a key step is to replace the original llm by a pt-pt llm, we leverage a recent pt-pt text-only decoder, glória @xcite , and conduct extensive experiments, in a zero-shot setting, on image caption and visual dialog tasks. ## related work most generative vision and language models consist of decoder-only transformers. @xcite showed that when trained with a lot of data, language models can generalize and solve new (unseen) tasks. this is very useful since although the training is expensive and requires a lot of data, once they are pre-trained, they can be applied to a myriad of tasks with reduced adaptation costs. llava @xcite takes advantage of this by creating a general vision and language model using a frozen llm as decoder and a frozen visual encoder to encode the images, training a linear layer that transforms the image embeddings into the llm embedding space. this simple linear transformation has the advantage of introducing a very small number of parameters to be learned, allowing for efficient large-scale training, while leveraging the generalization capabilities of the backbone text llm. different ways of mapping image embeddings to the llm token subspace have been tried, such as a q-former @xcite consisting of a query transformer that learns query embeddings that will interact with the image encoding through cross attention, and cogvlm @xcite where although the part of the llm that processes the text input will still be frozen, it trains the weights used to compute the queries, keys, and values relative to the image embeddings. fromage @xcite takes a step further by extra linear transformations that enable to model to generatively retrieve images/texts. this is accomplished by introducing a special retrieval token, that is then trained under a multimodal contrastive learning of cross-modal mappings. most of these models are in english or other high-resource languages. very recently, open european portuguese llms have been made available. in particular, glória @xcite is a european portuguese llm decoder based on gpt-neo @xcite -which approximates the gpt3 architecture -trained on a 35b token corpus, from a diverse set of domains, including web content, news pieces, encyclopedic knowledge, news articles, and dialogs. gervásio @xcite is another relevant european portuguese llm decoder which is based on a pre-trained llama 2 7b @xcite model, fine-tuned on portuguese instruction datasets, comprising around 83m tokens. regarding v&l approaches, literature is scarce. capivara (dos santos et al., 2023) trains a brazilian portuguese clip model, while performing data augmentation through image captioning and machine translation. in this work, and using recent developments in the llm pt-pt, we seek to narrow this gap, by introducing a european portuguese v&l model. ## pt-pt datasets for vision and language ai due to the lack of european portuguese v&l datasets, we embraced the task of translating core vision and language datasets from english into european portuguese. given the size of existing datasets (millions scale), translating the datasets with human experts would be too costly, hence we considered three distinct automatic machine translation models: first, we considered a) madlad-400 @xcite , a model trained on a 3t token dataset based on common-crawl, created by google, covering text data from over 400 languages; b) narrativa foot_1 , which is an mbart-50 @xcite ) model fine-tuned on the opus-100 @xcite dataset for english to portuguese translation, c) deepl foot_2 a commercial translation service. we started by pre-assessing the performance of each of the three approaches, using a subset of cc3m, comprising both shorter and longer captions. given these observations, we translated the cc3m @xcite , mscoco @xcite , and visdial @xcite datasets, using deepl and madlad-400 @xcite . given the higher effectiveness of deepl, we will take them as the main datasets/benchmarks, and refer to them as cc3m pt-pt, mscoco pt-pt and visdial pt-pt, respectively. the cc3m pt-pt dataset was used as the pre-training dataset, and both mscoco pt-pt and visdial pt-pt were used for benchmarking retrieval, image-captioning and visual dialog tasks. ## method in this section we present v-glória, an european portuguese large v&l model, capable of flexibly interleaving the two modalities, images and text, and therefore generalize to different nlp and cv tasks such as multimodal retrieval, image captioning, and visual dialog. therefore, we adapt the fromage model (koh et al., 2023) architecture, which leverages a text llm and adds a set of projection layers to align images with the llm input subspace, and support generative retrieval. specifically, it allows us to use an european portuguese llm, that will be frozen during training, with lightweight training strategies aimed at equip v-glória with visual and linguistic reasoning capabilities. ## experimental setup we assess the performance of our model in both image retrieval and image-and-text generation tasks. the models were trained on the cc3m pt-pt dataset, originally comprising 3.3 million imagetext pairs, which after filtering out missing and corrupted images resulted in a total of 2.7m samples. we consider both glória 1.3b foot_4 and gervásio 7b foot_5 as the pt-pt llm backbones. multimodal retrieval and image captioning are evaluated in both the cc3m pt-pt (full-shot) and mscoco pt-pt (zero-shot) evaluation sets. models are also evaluated in the visual dialog task @xcite , in a zero-shot setting. to establish a comparison between english and european portuguese, we consider the architectural twin of glória 1.3b, gptneo 1.3b @xcite , an english-only llm. training details. for training, we set a batch size of 180 and train for a total of 20000 iterations, taking about 24 hours on 1x nvidia a100 40gb gpu. we used the adam optimizer (kingma and ba, 2015) with a learning rate of 0.0003 and a warmup of 100 steps. the loss weight λ c and λ r are set to 1, the visual prefix length of @xmath0. as for the embedding dimensions, we set the retrieval embedding dimension to @xmath1, and model inner embedding dimension to @xmath2. as most of the model parameters are frozen, our method is memory and compute-efficient: we backpropagate through the frozen llm and visual model but only compute gradient updates for the 3 trainable linear mappings and [ret] embedding. ## results and discussion in this section, we discuss the experimental results in the image captioning and cross-modal retrieval tasks. we start by evaluating our model in crossmodal retrieval, in both image to text (i2t) and text to image (t2i) settings, and then in image captioning. we follow related work, and for crossmodal retrieval experiments adopt as metrics recall@5 (r@5), and recall@10 (r@10), and for image captioning bleu and meteor. finally, we consider the challenging task of visual dialog, in a zero-shot setting. for the three tasks, we follow the established task evaluation protocols. ## conclusions in this paper, we proposed a methodology to efficiently customize a vision and language llm to european portuguese. in particular, we introduced v-glória, the first european-portuguese vision and language model, capable of addressing multimodal tasks such as retrieval, image captioning, and visual dialogs illustrated in figure 3 . experiments, leveraging current best performing open pt-pt llms as backbones, reveal performances that are competitive with the english counterpart setting (i.e. english pre-training and benchmarks), on these tasks. v-glória demonstrated to be capable of generalizing to unseen data, especially in multimodal retrieval. for more challenging tasks, such as visual dialog, the proposed approach is still not on par with english models. however, we believe that as better pt-pt models arise, including instruction-tuned ones, the performance gap can be narrowed down by employed our devised customization methodology, and leveraging our contributed pt-pt data resources. we will release the pt-pt high-quality translations of the most popular v&l datasets to foster research in this area.