Stop Pre-Training: Adapt Visual-Language Models to Unseen Languages (2023)

Abstract: Vision-Language Pre-training (VLP) has advanced the performance of many vision-language tasks, such as image-text retrieval, visual entailment, and visual reasoning. The pre-training mostly utilizes lexical databases and image queries in English. Previous work has demonstrated that the pre-training in English does not transfer well to other languages in a zero-shot setting. However, multilingual pre-trained language models (MPLM) have excelled at a variety of single-modal language tasks. In this paper, we propose a simple yet efficient approach to adapt VLP to unseen languages using MPLM.We utilize a cross-lingual contextualised token embeddings alignment approach to train text encoders for non-English languages. Our approach does not require image input and primarily uses machine translation, eliminating the need for target language data. Our evaluation across three distinct tasks (image-text retrieval, visual entailment, and natural language visual reasoning) demonstrates that this approach outperforms the state-of-the-art multilingual vision-language models without requiring large parallel corpora. Our code is available at https://github.com/Yasminekaroui/CliCoTea .

## introduction inspired by the recent advancements in language model pre-training, vision-language pre-trained models (vlpms) have demonstrated state-of-theart performance across a wide range of visionlanguage (vl) tasks such as text-to-image retrieval, visual reasoning, visual entailment, and visual qa @xcite @xcite . however, extending vlpms to multilingual scenarios is still challenging. on one hand, the majority of these models are trained on monolingual (english) corpora and thus cannot perform well for other languages. on the other hand, the multilingual pre-trained language models @xcite the basketball player shoots a three pointer pemain bola basket menembakkan tembakan tiga angka 2018; @xcite cannot handle vision data (e.g., images or videos) directly. lately, there have been attempts (m 3 p, nuniter, uc 2 ) to pivot on images or english texts to align multilingual representations with vision features @xcite @xcite . however, a recent benchmark on multilingual multimodal pretraining (iglue) @xcite shows that although these models achieve promising zeroshot cross-lingual transfer performance on some vl tasks, they still fall short in comparison to the "translate-test" baseline (using an english-only vlpm on the translations of the text examples). a more recent work (cclm) achieves promising performance on the iglue benchmark by exploiting massive parallel text and image-text corpora to pre-train a vl model @xcite . this approach is motivated by a key observation that multilingual and multimodal pre-training essentially achieves the same goal of aligning two different views of the same object into a common semantic space. although this framework performs well on the iglue benchmark, it requires a large amount of parallel data. its pre-training phase relies on 19m multilingual parallel sentence pairs extracted from wikimatrix @xcite , jointly trained with 4 million image-text pairs in multiple languages. in this work, we are proposing a simple yet efficient way to adapt vlp models to unseen languages without requiring large parallel corpora. we propose to align a vlpm monolingual text encoder (achieving start-of-the-art performance on english downstream vl tasks) with a multilingual pre-trained language model (e.g., mbert), using only small in-domain parallel text corpus. the recent progress in neural machine translation (nmt) has enabled us to create such a parallel corpus from automatically translating the data from english to any other language, even for lowresource languages (i.e., swahili). however, since our approach relies on token alignment, it is robust to errors made by nmt. our zero-shot evaluation across three of the four iglue tasks shows that the proposed method achieves state-of-the-art results while using small set of in-domain parallel sentences. the key steps of our approach are illustrated in figure 1 . ## conclusion in this paper, we present clicotea an approach for adapting vision-language pre-trained models to unseen languages. unlike other approaches that rely on an expensive pre-training phase (both in terms of data and computation), our approach adapts the contextualised token embeddings of a multilingual pre-trained language model by aligning them with the contextualised token embeddings of the vlpm text encoder. by aligning albef text encoder with mbert, we show that clicotea outperforms cclm, which exploits massive parallel text and image-text corpora. clicotea achieves start-of-the-art performance on visual entailment and visual reasoning, with an increase of almost 5% on visual entailment. it also demonstrates its effectiveness, especially for low-resource languages, as it does not require large corpora to do the adaptation.