UNIMO : Towards Unified-Modal Understanding and Generation via Cross-Modal Contrastive Learning (2021)

Abstract: Existed pre-training methods either focus on single-modal tasks or multi-modal tasks, and cannot effectively adapt to each other. They can only utilize single-modal data (i.e., text or image) or limited multi-modal data (i.e., image-text pairs). In this work, we propose a UNIfied-MOdal pre-training architecture, namely UNIMO, which can effectively adapt to both single-modal and multi-modal understanding and generation tasks. Large scale of free text corpus and image collections are utilized to improve the capability of visual and textual understanding, and cross-modal contrastive learning (CMCL) is leveraged to align the textual and visual information into a unified semantic space, over a corpus of image-text pairs augmented with related images and texts. With the help of rich non-paired single-modal data, our model is able to learn more generalizable representations, by allowing textual knowledge and visual knowledge to enhance each other in the unified semantic space. The experimental results show that UNIMO greatly improves the performance of several single-modal and multi-modal downstream tasks. Our code and pre-trained models are public at https://github.com/PaddlePaddle/Research/tree/master/NLP/UNIMO .

## introduction large-scale pre-training has drawn much attention in both the community of compute vision (cv) and natural language processing (nlp) due to its strong capability of generalization and efficient usage of large-scale data. firstly in cv, a series of models were designed and pre-trained on the large-scale dataset imagenet, such as alexnet @xcite , vgg (simonyan and * these authors contribute equally to this study and are listed with random order. who is standing behind the baseball player? any baseball game involves one or more umpires, who make rulings on the outcome of each play. at a minimum, one umpire will stand behind the catcher, to have a good view of the strike zone, and call balls and strikes. additional umpires may be stationed near the other bases … from wikipedia zisserman, 2014) and resnet @xcite , which effectively improved the capability of image recognition for numerous tasks. recent years have witnessed the burst of pre-training in nlp, such as bert @xcite , roberta @xcite , xlnet @xcite and unilm @xcite , which greatly improve the capabilities of language understanding and generation. however, the above researches focus on the singlemodal learning and can only be effectively used in single-modal (i.e., only text or image) scenarios. in order to adapt to multi-modal scenarios, a series of multi-modal pre-training methods were proposed and pre-trained on the corpus of image-text pairs, such as vilbert @xcite , visualbert @xcite and uniter @xcite , which greatly improve the ability to process multimodal information. however, these models can only utilize the limited corpus of image-text pairs and cannot be effectively adapted to single-modal scenarios @xcite . a smarter ai system should be able to process different modalities of information effectively. there are large scale of data in different modalities on the web, mainly textual and visual information. the textual knowledge and the visual knowledge img] [roi1] [roi2] [roi3] [roi4] [roi5] [roi6] [roin] image collections text corpus any baseball game i n v o l v e s o n e o r more umpires… at a m i n i m u m , o n e umpire will stand behind the catcher… ## unimo humans perceive the world through many modalities, such as sound, vision and language. even ## experimental settings in this section, we introduce the pre-training and finetuning experimental settings. ## results and analysis in this section, we report the evaluation results on both the multi-modal and single-modal tasks to show the adaptability and generalizability of unimo to different scenarios. we further make several ablation studies to validate that textual knowledge and visual knowledge can enhance each other in the unified semantic space. the visualization and case analysis of the model results are appended in appendix c. ## related work existing researches on pre-training can be mainly classified into two categories: single-modal pretraining and multi-modal pre-training. the singlemodal pre-training methods only focus on singlemodal tasks, while the multi-modal pre-training methods only focus on multi-modal tasks. single-modal pre-training the single-modal pre-training methods mainly consist of visual pretraining and language pre-training. most visual pre-training methods are based on the multi-layer cnn architecture such as vgg @xcite and resnet @xcite , and trained on the imagenet dataset. recently, contrastive self-supervised learning like simclr @xcite and moco @xcite also greatly improve the performance of visual representation learning. these pre-trained models only focus on visual tasks (e.g. image classification etc.), however, they cannot be used in textual or multimodal (i.e., with both text and image) tasks. the language pre-training methods based on the transformer architecture are also very popular in nlp models, such as gpt @xcite , bert @xcite , xlnet @xcite and bart @xcite . however, they mainly focus on textual tasks. they cannot effectively deal with the multi-modal tasks, such as image-text retrieval, image captioning, multimodal machine translation @xcite and visual dialog @xcite . multi-modal pre-training recently, multimodal pre-training methods have been more and more popular for solving the multi-modal tasks. all of them are trained on a corpus of image-text pairs, such as vilbert @xcite , visualbert @xcite , vl-bert @xcite , unicoder-vl @xcite and uniter @xcite . based on the multi-layer transformer network, they all employ the bert-like objectives to learn multi-modal representations from a concatenated-sequence of vision features and language embeddings. their architectures can be mainly classified into two categories: single-stream and two-stream. the two-stream methods, such as vilbert, utilize two single-modal transformer to process visual features and language embeddings respectively, and then learn their interactions based on a crossmodal transformer. the single-stream methods directly utilize a single transformer network to model both the visual features and the language embeddings. visualbert, vl-bert, unicoder-vl and uniter all utilize the single-stream architecture, which show that fusing cross-modal information early and freely by a single-stream network can achieve better performance. recently, several contrastive learning-based multi-modal pre-training methods have also been proposed. openai clip @xcite leverages large-scale image-text pairs to learn transferrable visual representations by image-text matching, which enables zero-shot transfer of the model to various visual classification tasks. wenlan @xcite further proposes a similar two-tower chinese multi-modal pre-training model and adapts moco @xcite to improve the contrastive cross-modal learning process. instead of extracting salient image regions by pre-trained object detection models like faster-rcnn @xcite , the end-to-end vision-language pre-training architecture soho @xcite proposes to jointly learn convolutional neural network (cnn) and transformer for cross-modal alignments from millions of image-text pairs. all existed multi-modal pre-training methods only focus on multi-modal tasks with both vision and language inputs. however, they cannot be effectively adapted to single-modal tasks. moreover, they can only utilize the limited corpus of image-text pairs. by contrast, our unified-modal pre-training method unimo can employ large volumes of text corpus and image collections to enhance each other, and can be effectively adapted to both textual and multi-modal scenarios. unimo also achieves the best performance on multi-modal tasks including image-text retrieval, visual entailment, vqa and image caption. ## conclusion in this work, we propose unimo, a unified-modal pre-training architecture to leverage the large scale of non-paired text corpus and image collections for cross-modal learning. we verify that unimo provides an effective way for textual knowledge and visual knowledge to mutually enhance each other in a unified semantic space, and unimo successfully adapts to both single-modal and multi-modal understanding and generation tasks. in this way, unimo outperforms previous methods on both the multi-modal and single-modal downstream tasks. in the future work, we will focus on end-to-end visual and language unified learning, and much larger scale of model size and data volumes.