m CLIP : Multilingual CLIP via Cross-lingual Transfer (2023)

Abstract: Large-scale vision-language pretrained (VLP) models like CLIP have shown remarkable performance on various downstream cross-modal tasks. However, they are usually biased towards English due to the lack of sufficient non-English image-text pairs. Existing multilingual VLP methods often learn retrieval-inefficient single-stream models by translation-augmented non-English image-text pairs. In this paper, we introduce mCLIP, a retrieval-efficient dual-stream multilingual VLP model, trained by aligning the CLIP model and a Multilingual Text Encoder (MTE) through a novel Triangle Cross-modal Knowledge Distillation (TriKD) method. It is parameter-efficient as only two light projectors on the top of them are updated during distillation. Furthermore, to enhance the token- and sentence-level multilingual representation of the MTE, we propose to train it with machine translation and contrastive learning jointly before the TriKD to provide a better initialization. Empirical results show that mCLIP achieves new state-of-the-art performance for both zero-shot and finetuned multilingual image-text retrieval task.

## introduction recently, large-scale dual-stream vision-language pretrained (vlp) models, such as clip @xcite , align @xcite and their variants @xcite @xcite , have shown remarkable performance on various downstream multimodal tasks. these models use separate encoders for the images and texts, and allow efficient inference in the image-text retrieval task because the image or text features can be computed offline. however, most current vlp models are biased toward english, due to the lack of sufficient high-quality multilingual multimodal datasets for direct large-scale pretraining. despite the lack of sufficient non-english imagetext pairs, previous methods attempt to create wordlevel code-switched image-text pairs by looking up bilingual dictionaries @xcite , or sentencelevel augmented multilingual image-text pairs by translating the english text to other languages (i.e, translate-train pipeline) @xcite . then each image and its paired text are concatenated as a single sequence to train a single-stream transformer-based model. despite the good performance of these models @xcite , they are less efficient than dual-stream models on large-scale image-text retrieval tasks, as the data from both modalities are intertwined to compute the self-attention and the unimodal features can not be pre-computed. instead of creating word-level or sentence-level multilingual imagetext pairs, mural @xcite extends the align @xcite model with multilinguality by an additional text-text contrastive loss among hundreds of languages. however, mural is trained from scratch and requires large-scale training data with high computation cost to obtain strong performance on multilingual cross-modal retrieval tasks. to tackle the aforementioned problems, we propose the triangle cross-modal knowledge distillation (trikd) to learn a dual-stream multilingual vlp model mclip, which learns triangle alignment among the pretrained clip's image encoder, clip's text encoder and a pretrained multilingual text encoder (mte) through knowledge distillation. specifically, to avoid catastrophic forgetting of the knowledge already learned in the pretrained clip and mte, they are kept frozen. the triangle alignment is achieved by adjusting a linear projector on top of clip and a shallow transformerbased x-projector on top of the mte. since the xlm-r @xcite used for initializing the mte has unsatisfactory performance when directly used for retrieval tasks @xcite , before performing the trikd, we propose to enhance the mte via both the machine translation task and a contrastive loss to improve the token-and sentence-level multilingual representation. the proposed mclip is both parameter-and computation-efficient as only the projectors are trained, which accounts for only 3% of the total parameters of mclip. empirical results of zeroshot and finetuned multilingual image-text retrieval on mscoco @xcite and multi30k @xcite show that the proposed mclip achieves better performance while being much more efficient in inference than single-stream baselines or using less training data than mural. the proposed method can also be extended to train a multilingual vlp based on a unimodal image encoder and the mte, 89.4% performance retained. foot_0 2 related work multilingual vlp models. monolingual visionlanguage pretrained (vlp) models @xcite @xcite trained with large-scale image-text pairs have shown remarkable performance on various downstream tasks like image-text retrieval. recently, some attempts have extended vlp models to the multilingual scenario. the first line of work applies the translation method to create multilingual image-text pairs and then concatenates the multilingual text with its paired image as a single sequential input to a single-stream transformerbased encoder. for instance, m3p @xcite constructs a multilingual code-switched text by randomly replacing english words with translations of other languages, and uc2 @xcite directly translates a whole sentence into other languages. however, these single-stream models are inefficient for the image-text retrieval task as unimodal features cannot be pre-computed beforehand. mural @xcite directly trains from scratch with both augmented multilingual imagetext pairs and parallel text corpus, which is expensive in both data and computation. besides retrieval tasks, recent pali @xcite and ernie-unix2 @xcite use the encoder-decoder architectures for multilingual multimodal generation tasks. in this paper, we introduce a data-and parameter-efficient knowledge distillation method to train a dual-stream multilingual vlp model by aligning a frozen english vlp and a frozen mte. knowledge distillation. knowledge distillation @xcite is firstly proposed for model compression. the knowledge in the output logits of a large teacher model can be transferred to a smaller student model without significant performance degradation. besides the logits, the hidden states and attention outputs can also be used for knowledge distillation @xcite . recently, @xcite propose to distill knowledge with contrastive learning, which maximizes the mutual information between the teacher and student models. for multimodal models, @xcite propose to train a dualstream vlp model with the knowledge distilled from a single-stream model for faster inference. furthermore, vlkd @xcite augments a dual-stream vlp model with a pretrained language model via vision-language knowledge distillation, enabling the multimodal generation ability without hurting the original nlp ability. however, to the best of our knowledge, knowledge distillation has not been studied for training multilingual vlp models, for which efficiency is an important factor due to the data scarcity issue. in this paper, we introduce a novel triangle cross-modal knowledge distillation method to efficiently align a multilingual text encoder to the multimodal space of a pretrained dual-stream vlp model. ## method in this section, we first introduce the architecture of mclip in section 3.1. it extends the monolingual vlp model clip to a multilingual one by aligning clip and a multilingual text encoder (mte) to a shared space, through a novel triangle cross-modal knowledge distillation (trikd) using english image-text pairs (section 3.2). the performance of mclip on non-english image-text retrieval is highly dependent on the quality of the multilingual representation of the mte. thus in section 3.3, we propose to first improve the token-and sentence-level cross-lingual representation of the mte with the neural machine translation (nmt) task and contrastive learning (ctl). ## analysis of the representations the training objectives of contrastive learning encourage the positive samples to stay closer (i.e., alignment) while the negative samples to scatter on the hypersphere (i.e., uniformity) @xcite . similarly, a desired multilingual and multimodal model should also learn good alignment between images and multilingual texts, as well as uniform representations within each modality. we analyze the quality of the learned representations with the uniformity and alignment scores introduced in @xcite . the alignment score ℓ @xmath0.e, h i i , h x i for the i-th image-text pair), while the uniformity score measures how uniformly the representations are distributed: from ## conclusion in this paper, we introduce mclip, a novel multilingual vision-language pretrained model which aligns clip and an enhanced multilingual text encoder through triangle cross-modal knowledge distillation. this distillation method is both parameterefficient with only 3% of the total parameters of mclip trained, and data-efficient with only english image-text pairs required. the performance of mclip can be further improved with more parallel text corpus from more languages and mul-tilingual image-text pairs from the translate-train pipeline. empirical results show that the proposed mclip+ achieves state-of-the-art performance in multilingual image-text retrieval tasks.