Multilingual Multimodal Pre-training for Zero-Shot Cross-Lingual Transfer of Vision-Language Models (2021)

Abstract: This paper studies zero-shot cross-lingual transfer of vision-language models. Specifically, we focus on multilingual text-to-video search and propose a Transformer-based model that learns contextual multilingual multimodal embeddings. Under a zero-shot setting, we empirically demonstrate that performance degrades significantly when we query the multilingual text-video model with non-English sentences. To address this problem, we introduce a multilingual multimodal pre-training strategy, and collect a new multilingual instructional video dataset (Multi-HowTo100M) for pre-training. Experiments on VTT show that our method significantly improves video search in non-English languages without additional annotations. Furthermore, when multilingual annotations are available, our method outperforms recent baselines by a large margin in multilingual text-to-video search on VTT and VATEX; as well as in multilingual text-to-image search on Multi30K. Our model and Multi-HowTo100M is available at http://github.com/berniebear/Multi-HT100M .

## introduction one of the key challenges at the intersection of computer vision (cv) and natural language processing (nlp) is building versatile vision-language models that not only work in english, but in all of the world's approximately 7,000 languages. since collecting and annotating task-specific parallel multimodal data in all languages is impractical, a framework that makes vision-language models generalize across languages is highly desirable. one technique that has shown promise to greatly improve the applicability of nlp models to new languages is zero-shot cross-lingual transfer, where models trained on a source language are applied * equal contribution. for cross-lingual transfer of vision-language models, the visual information is clearly an essential element. to this end, we make an important yet under-explored step to incorporate visual-textual relationships for improving multilingual models @xcite . while spoken languages could be different, all humans share similar vision systems, and many visual concepts can be understood universally @xcite . for example, while is termed "cat" for an english speaker and "chat" for a french speaker; they understand similarly. we leverage this observation to learn to associate sentences in different languages with visual concepts for promoting cross-lingual transfer of visionlanguage models. in this work, we focus on multilingual text-tovideo search tasks and propose a transformerbased video-text model to learn contextual mul-tilingual multimodal representations. our vanilla model yields state-of-the-art performance in multilingual @xmath0. however, under the zero-shot setting, rather surprisingly, there is a significant performance gap between english and non-english queries (see §5.5 for details). to resolve this problem, motivated by recent advances in large-scale language model @xcite and multimodal pre-training @xcite @xcite , we propose a multilingual multimodal pre-training (mmp) strategy to exploit the weak supervision from large-scale multilingual text-video data. we construct the multilingual-howto100m dataset, that extends the english howto100m @xcite dataset to contain subtitles in 9 languages for 1.2 million instructional videos. our method has two important benefits. first, compared to pre-training on english-video data only, pre-training on multilingual text-video data exploits the additional supervision from a variety of languages, and therefore, enhances the search performance on an individual language. second, by exploiting the visual data as an implicit "pivot" at scale, our methods learns better alignments in the multilingual multimodal embedding space (e.g., "cat"--"chat"), which leads to improvement in zero-shot cross-lingual transfer (e.g., from "cat"to "chat"-) of vision-language models. in our experiments on vtt @xcite ) and vatex @xcite , our method yields state-of-the-art @xmath1. for zero-shot cross-lingual transfer, the proposed multilingual multimodal pre-training improves english-video pre-training by 2 ∼ 2.5 in average r@1 across 9 languages. additionally, when trained with in-domain multilingual annotations as other baselines, our method outperforms them by a large margin in multilingual @xmath2. to summarize, we make the following contributions: (1) we propose a transformer-based videotext model that learns contextual multilingual multimodal representations ( §3.1). (2) we empirically demonstrate that vision-language models, unlike nlp models, have limited zero-shot cross-lingual transferrability. ( §5.5). (3) we introduce the multilingual multimodal pre-training strategy and construct a new multi-howto100m dataset ( §4) for pre-training to improve zero-shot cross-lingual capability of vision-language models. (4) we demonstrate the effectiveness of our approach, by achieving state-of-the-art multilingual @xmath3.5) and fully supervised setup ( §5.6). ## related work cross-lingual representations. early work on learning non-contextual cross-lingual representations used either parallel corpora @xcite or a bilingual dictionary to learn a transformation @xcite . later approaches reduced the amount of supervision using self-training @xcite . with the advances in monolingual transfer learning @xcite @xcite , multilingual extensions of pre-trained encoders have been proven effective in learning deep contextual cross-lingual representations @xcite @xcite @xcite . we extend prior work to incorporate visual context. video-text representations. the howto100m dataset @xcite has attracted significant interest in leveraging multimodal pre-training for @xmath4, captioning @xcite , and unsupervised translation via image-based @xcite @xcite alignment. this work studies a challenging and unexplored task: zero-shot cross-lingual transfer of vision-language models. unlike prior image/video-text work that utilizes rnn @xcite @xcite and inter-modal contrastive objectives @xcite @xcite , we employ transformers to learn contextual multilingual multimodal representations and uniquely models cross-lingual instances. moreover, we build multi-howto100m, the largest text-video dataset for multilingual multimodal pre-training. cross-lingual transfer. cross-lingual transfer has proven effective in many nlp tasks including dependency parsing @xcite , named entity recognition @xcite , sentiment analysis @xcite , document classification @xcite , and question an- ## method we consider the problem of learning multilingual multimodal representations from a corpus c of video-text pairs {(x i , v i )} c @xmath5, where v i is a video clip and x i is its corresponding text (caption or transcription) that is written in one of k languages. our goal is to learn a shared multilingual text encoder c @xmath6, both of which project the input to a shared ddimensional embedding space c v , c @xmath7, where semantically similar instances (i.e., paired (x i , v i )) are closer to each other than the dissimilar ones (i.e., (x i , v j ), @xmath8. in the following, we denote a batch of multilingual text-video samples as @xmath9, v i )} b @xmath10. ## the multilingual howto100m dataset as large-scale pre-training has been shown important in recent nlp and vision-language models, we construct the multilingual howto100m dataset (multi-howto100m) to facilitate research in multilingual multimodal learning. the original howto100m @xcite dataset is a large-scale video collection of 1.2 million instructional videos (around 138 million clips/segments) on youtube, along with their automatic speech recognition (asr) transcriptions as the subtitles. for each video in howto100m, we crawl and collect the multilingual subtitles provided by youtube, which either consist of user-generated subtitles or those generated by google asr and translate in the absence of user-generated ones. essentially, we collect video subtitles in 9 languages: english (en), german (de), french (fr), russian (ru), spanish (es), czech (cz), swahili (sw), chinese (zh), vietnamese (vi). at the time of dataset collection (may 2020), there are 1.1 million videos available, each with subtitles in 7-9 languages. the video length ranges from 1 minute to more than 20 minutes. we utilize multi-howto100m for multilingual multimodal pre-training to exploit the weak supervision from large-scale multilingual text-video data. in fig. 2 , we provide a visualization of few instances sampled in multi-howto100m with the corresponding video frame, timestamp, and transcriptions in different languages. please refer to appendix for more details and dataset statistics. ## experiment in this section, we first describe our experimental setup ( §5.1-5.3). in §5.4, we conduct ablation studies to validate the effectiveness of proposed multilingual text-video model . with the best models at hand, we investigate their zero-shot cross-lingual transferability in §5.5, where we showcase that the proposed multilingual multimodal pre-training serves as the key facilitator. we then verify the superior @xmath11, multilingual, and cross-modality settings in §5.6. ## conclusion we have presented a multilingual multimodal pretraining (mmp) strategy, the multi-howto100m dataset, and a transformer-based text-video model for learning contextual multilingual multimodal representations. the results in this paper have convincingly demonstrated that mmp is an essential ingredient for zero-shot cross-lingual transfer of vision-language models. meanwhile, there are many remaining challenges, such as resolving the performance gap between zero-shot and training with in-domain non-english annotations; as well as techniques to transfer varieties of vision-language models (e.g., vqa @xcite , tvqa @xcite ) or visually-enhanced nlp models such as unsupervised multimodal machine translation @xcite . we believe the proposed methodology, and the corresponding resources we release, will be an important first step towards spurring more research in this direction.