Cross-View Language Modeling: Towards Unified Cross-Lingual Cross-Modal Pre-training (2023)

Abstract: In this paper, we introduce Cross-View Language Modeling, a simple and effective pre-training framework that unifies cross-lingual and cross-modal pre-training with shared architectures and objectives. Our approach is motivated by a key observation that cross-lingual and cross-modal pre-training share the same goal of aligning two different views of the same object into a common semantic space. To this end, the cross-view language modeling framework considers both multi-modal data (i.e., image-caption pairs) and multi-lingual data (i.e., parallel sentence pairs) as two different views of the same object, and trains the model to align the two views by maximizing the mutual information between them with conditional masked language modeling and contrastive learning. We pre-train CCLM, a Cross-lingual Cross-modal Language Model, with the cross-view language modeling framework. Empirical results on IGLUE, a multi-lingual multi-modal benchmark, and two multi-lingual image-text retrieval datasets show that while conceptually simpler, CCLM significantly outperforms the prior state-of-the-art with an average absolute improvement of over 10%. Moreover, CCLM is the first multi-lingual multi-modal pre-trained model that surpasses the translate-test performance of representative English vision-language models by zero-shot cross-lingual transfer.

## introduction recently, the tremendous success of selfsupervised language model pre-training @xcite @xcite @xcite @xcite @xcite has been expanded to the * equal contribution. work done at bytedance. † correspondence to: zengyan.yanne@bytedance.com 1 the code and pre-trained models are available at https: //github.com/zengyan-97/cclm . while appealing, multi-lingual multi-modal pretraining has its own challenges. unlike multilingual pre-training and multi-modal pre-training where relatively large amount of parallel data is available, there exists only a few multi-lingual multi-modal corpora and their language coverage is also limited. two pioneering works, m 3 p @xcite and uc 2 @xcite , propose to pivot either on english texts or images to align multi-lingual multi-modal representations. both of them introduce a number of new objectives to make use of the anchor for alignment. however, a recent benchmark on multi-lingual multi-modal pre-training @xcite reveals that these multi-lingual multi-modal pre-trained models are still falling short: while achieving seemingly promising zero-shot cross-lingual transfer performance on some vision-and-language tasks, they still significantly under-perform "translatetest", a simple baseline which translates the test examples into english and uses an english-only vision-language model for inference. this prevents existing multi-lingual multi-modal models to be applicable in real-world applications. in contrast, multi-lingual pre-trained models such as xlm-r @xcite significantly outperforms the translate-test baseline in most languages and is widely used in practical applications. this paper aims to fully exploit the potential of multi-lingual multi-modal pre-training. we point out two major limitation of current state-of-thearts. first, existing methods do not exploit parallel text corpora, which can be easily collected and are abundant for many language pairs. instead, m 3 p performs masked language modeling with monolingual texts in different languages for multi-lingual alignment. however, parallel texts are shown to be more helpful according to multi-lingual pretraining literature @xcite . second, a number of new pre-training objectives involving specific architecture changes and different input-output formats are introduced for english or image pivoting, making it non-trivial to combine them together for better performance and scale to larger data. in this work, we argue that multi-lingual and multi-modal pre-training are essentially achieving the same goal of aligning two different views of a same object into a common semantic space. therefore, we believe these two seemingly different strategies can be combined into a unified framework. to this end, we introduce cross-view language modeling, a simple and effective framework that unifies cross-lingual and cross-modal pretraining with shared architecture and objectives. specifically, we consider both multi-modal data (i.e., image-caption pairs) and multi-lingual data (i.e., parallel sentence pairs) as pairs of two different views of the same object. with either multimodal or multi-lingual data as input, we encode the two views with transformer models and then fuse their representations with a cross-attention transformer model shared for both cross-modal and cross-lingual fusion. we train the model to align the two views into a common semantic space by maximizing the mutual information between them with a conditional masked language modeling objective, a contrastive learning objective, and a matching objective. in this way, the cross-view language modeling framework unifies english pivoting and image pivoting schemes seamlessly and makes the best of both worlds. to evaluate the effectiveness of our approach, we pre-train cclm, a cross-lingual cross-modal language model, with the proposed cross-view language modeling framework. experimental results show that cclm significantly outperforms prior state-of-the-art with an averaged absolute improvement of over 10% and 30% on multi-lingual visionlanguage understanding and retrieval tasks in terms of accuracy and r@1 on iglue @xcite , a recently released multi-lingual multimodal benchmark. notably, cclm is the first multi-lingual vision-language model that surpasses the "translate-test" performance of mono-lingual vision-language models via zero-shot cross-lingual transfer, which we believe is a crucial step towards practical multi-lingual multi-modal pre-training. since previous work used different pre-training datasets, making direct comparison difficult, we also conduct an in-depth ablation study to investigate the contribution of different parts in our framework. the results show that use of parallel sentence pairs helps to fully exploit the potential of language pivoting for multi-lingual multi-modal pre-training and also confirm the importance of unified architectures and objectives in cclm. ## conclusion in this paper, we introduce cross-view language modeling, a simple and effective framework that unifies cross-lingual and cross-modal pre-training. it considers cross-lingual and cross-modal pretraining as the same procedure of aligning the representation of two different views of the same object, thus using shared model architectures and training objectives for multi-lingual multi-modal pretraining. we train cclm with the proposed framework and show that it advances the state-of-the-art on all downstream multi-lingual vision-language tasks by a large margin. moreover, it surpasses the translate-test baseline for the first time, demonstrating the potential of multi-lingual multi-modal pre-training. furthermore, the experimental results also confirm that the proposed framework is scalable to massive data and larger model sizes. we believe our model will become a foundation for future multi-lingual multi-modal research and serve as a strong baseline. moreover, the cross-view language modeling framework also has the potential of unifying more modalities such as audio and video with the same architectures and objectives. we leave this for future work.