A lt CLIP : Altering the Language Encoder in CLIP for Extended Language Capabilities (2023)

Abstract: CLIP (Contrastive Language–Image Pretraining) is an English multimodal representation model learned from a massive amount of English text-image pairs and has achieved great success in various downstream tasks, including image classification, text-to-image retrieval, and image generation. When extending CLIP to other languages, the major problem is the lack of good-quality text-image pairs. In this work, we present AltCLIP, a simple and low-resource method to build a strong multilingual multimodal representation model. Instead of training a model from scratch on multilingual text-image pairs, we take the original CLIP model trained on English text-image pairs and alter its text encoder with a pre-trained multilingual text encoder (XLM-R). We then align text and image representations by a two-stage training schema consisting of teacher learning and contrastive learning. Our method utilizes the existence of rich parallel text data and pre-trained multilingual language models. We present extensive experimental evaluations to demonstrate the effectiveness of our proposed method. Our model sets new state-of-the-art zero-shot performances on a wide range of tasks in multilingual multimodal benchmarks, including ImageNet-CN/IT/JA/KO serials, Flicker30k-CN, COCO-CN, Multi30k, and XTD. Further, our model outperforms the original CLIP model on zero-shot cross-modal retrieval, Image Classification in the Wild (ICinW) tasks, and CLIP Benchmark. We plan to open-source our code, pre-trained model weights, and evaluation toolkits of multilingual multimodal tasks, to facilitate research on multilingual multimodal representation learning.

## introduction learning a good representation in a joint space for vision and language has been a long pur-suit in the research of artificial intelligence (ai). recently, the milestone work of clip @xcite from openai demonstrated impressive zero-shot performances across a number of tasks such as image classification on imagenet @xcite , image-to-text and text-to-image retrieval on flicker-30k @xcite and mscoco @xcite . there has been the pursuit of building contrastive language-image models in other languages such as italian @xcite , korean @xcite , chinese @xcite @xcite @xcite or in a cross-lingual and multilingual setting @xcite . training a good language-image representation model often requires a huge amount of text-image pairs and vast computational resources. for instance, clip used 400m text-image pairs, and taiyi @xcite , a recently proposed chinese model, used 123m text-image pairs. to alleviate these problems, several works manage to take advantage of the existing text-image representation clip and extend its language capabilities to other languages @xcite @xcite .cn-clip @xcite aligns a new chinese text encoder to the clip vision encoder through 200m chinese text-image pairs. more recently, m-clip @xcite proposed to use teacher learning (a.k.a. knowledge distillation) on the text encoder of the clip model to learn a multilingual text-image representation model. this method only uses machine-translated data from english to a target language, without text-image pairs. however, existing works in the cross-lingual or multilingual setting mainly focus on the model's retrieval performance and ignore their generalization ability. the dataset to evaluate retrieval performance is often small, e.g., 1, 000 images in test sets for flickr-30k. the retrieval performance fluctuates acutely with the change in training data distribution. although current methods achieve good performance in retrieval, these methods often do not perform well on the imagenet classification tasks. the ability to accurately predict images over 1, 000 classes often indicates better generalization ability of the model. to address the aforementioned problems, we propose a multilingual model named alter ego clip (altclip) which achieved strong performances on imagenet and multimodal retrieval tasks across languages. our proposed method altclip learns a multilingual text-image representation under a twostage framework (see figure 1 for an overview). in the first stage, we use teacher learning on parallel text to distill the knowledge learned from clip and align different languages and images. in the second stage, we further improve the alignment of text and image via contrastive learning @xcite on a moderate amount of multilingual text-image pairs. we employ this method to train a multilingual vision-language model that supports nine languages which we call altclip m 9 . we present an extensive experimental comparison over a variety of benchmarks and baseline methods, to demonstrate the effectiveness of our method. we show that using recall-based parallel text data in teacher learning can learn well-aligned text-image representation in both english and extended languages, while contrastive learning with text-image pairs effectively aligns the multilingual language model to the clip vision encoder. the model trained by this two-step training strategy results in a very strong performance on a broad range of multilingual multimodal benchmarks, including the original english multimodal benchmarks studied in clip @xcite . altclip m 9 sets new state-of-the-art results on multilingual image classification and retrieval tasks. furthermore, altclip m 9 achieves superior cross-modal performances in chinese, korean, japanese, and italian compared to methods trained from scratch with large-scale text-image pairs. lastly, we apply altclip m 9 to the task of text-to-image generations @xcite to show that it enables high-quality image generation from prompts in different languages. xlm-r @xcite is a multilingual language model that achieves strong performances on a wide range of cross-lingual tasks. in our work, we use the xlm-r model as the underlying text encoder and align it with the image encoder trained in clip, to achieve competitive performances on cross-lingual and cross-modality tasks. knowledge distillation. in knowledge distillation, the teacher-student architecture is a generic carrier to form knowledge transfer. the model capacity gap between a large deep neural network and a small student neural network can degrade knowledge transfer. @xcite . to effectively transfer knowledge to student networks, a variety of methods have been proposed for a controlled reduction of the model complexity @xcite @xcite . in this work, we use a multilingual model xlm-r as a student model for effectively transferring multilingual knowledge. ## methodology we propose a two-stage method to learn a multilingual multimodal representation model. in the first stage, we follow the work of @xcite to use teacher learning to learn a multilingual text encoder from the clip text encoder. in this step, no image is needed in training and only language parallel data is used. in the second stage, we use text-image pairs to further fine-tune the model from contrastive learning. our overall training procedure is summarized in figure 1 . ## experiments we present experimental results in this section. in section 5.1, we introduce the datasets and metrics used. we comprehensively validate our model through multilingual multimodal benchmarks in section 5.2. in section 5.3, we conduct an ablation study on the effects of various design choices in teacher learning and contrastive learning. finally, in section 5.4, we apply altclip to textimage generation, and show that our model is capable to align text in different languages. ## conclusion in this work, we propose an effective two-stage training method for learning multilingual multimodal representation models, through teacher learning and contrastive learning. the effectiveness is demonstrated through extensive experiments on a wide range of tasks in multilingual multimodal benchmarks. altclip m 9 outperforms the original clip model on many tasks in english and sets new state-of-the-art zero-shot results on multiple image classification tasks in chinese/korean/italian/japanese and multilingual re-trieval tasks. meanwhile, our method is highly data-efficient, which consumes only around 1% text-image pairs compared to the hundreds of millions of text-image pairs used by prior work on vision-language pretraining models.