UNIMO -2: End-to-End Unified Vision-Language Grounded Learning (2022)

Abstract: Vision-Language Pre-training (VLP) has achieved impressive performance on various cross-modal downstream tasks. However, most existing methods can only learn from aligned image-caption data and rely heavily on expensive regional features, which greatly limits their scalability and performance. In this paper, we propose an end-to-end unified-modal pre-training framework, namely UNIMO-2, for joint learning on both aligned image-caption data and unaligned image-only and text-only corpus. We build a unified Transformer model to jointly learn visual representations, textual representations and semantic alignment between images and texts. In particular, we propose to conduct grounded learning on both images and texts via a sharing grounded space, which helps bridge unaligned images and texts, and align the visual and textual semantic spaces on different types of corpora. The experiments show that our grounded learning method can improve textual and visual semantic alignment for improving performance on various cross-modal tasks. Moreover, benefiting from effective joint modeling of different types of corpora, our model also achieves impressive performance on single-modal visual and textual tasks. Our code and models are public at the UNIMO project page https://unimo-ptm.github.io/ .

## introduction large-scale pre-training has drawn much attention in the community of computer vision (cv), natural language processing (nlp) and multi-modal (mm) due to its strong capability of generalization and efficient usage of large-scale data. however, in the existing literature, the work on vision, language and vision-language representation learning are mostly studied separately with different training data sources. in the vision domain, pre-training on large-scale image corpus such as imagenet @xcite , openimages @xcite and jft-300m @xcite ) has proven to be critical for learning transferring visual representation for various downstream tasks. in nlp, pre-training on easily-accessible unannotated text corpora greatly improves the capabilities of language understanding and generation @xcite @xcite . pre-training has also become the de-facto approach in vision-language modeling @xcite @xcite @xcite . however, existing vlp methods require a massive amount of aligned image-text pairs which are costly to collect and hard to scale up. the large volumes of image corpus in cv and text corpus in nlp cannot be effectively utilized. thus, the scalability and performance upper limit of existing vlp methods are largely restricted. as they only learn joint vision-language representations on image-text pairs, they are also difficult to be effectively adapted to visual and textual tasks @xcite . to address the limitations, we propose a new end-to-end unified-modal pre-training framework, namely unimo-2, for joint learning on various types of corpora, including images, texts, and image-caption pairs. specifically, we build a unified transformer model to jointly learn visual representations, textual representations, and cross-modal alignment from the three types of corpora. both the visual and textual representations are learned end-to-end from raw images and textual sentences. combining a large number of unaligned images and texts is not only expected to improve the performance of joint vision-language tasks, but also improve the scalability of adapting to single-modal visual and textual tasks. however, it is challenging to bridge unaligned images and texts and effectively align the visual and textual semantic spaces on different types of corpora. only a few works have attempted to bridge unaligned images and texts by leveraging object tags from an pre-trained object detector as "anchor points" @xcite . however, they all rely heavily on expensive object-centric visual feature extraction, thus facing the problems of limited visual expressive power and computation inefficiency. in this paper, in order to bridge the unpaired image and text corpora and align the visual and textual semantic spaces end-to-end, we propose to conduct grounded learning on images, texts, and image-text pairs via a sharing grounded space. specifically, we introduce a grounded dictionary shared by images and texts, which represents vision-language grounded semantics. to learn the grounded dictionary, we apply vector quantization on both visual and textual representations to group image patches and text tokens with similar semantics into grounded tokens. furthermore, we design a grounded transformer architecture to let the visual and textual information exchanged by the grounded tokens, which not only facilitates grounded dictionary learning, but also improves cross-modal alignment. our grounded learning method can help bridge the textual and visual semantic spaces on unpaired image and text corpora to improve cross-modal fusion on different types of corpora. we evaluate unimo-2 on a variety of representative vision-language understanding and generation tasks, including image/text retrieval, visual question answering, visual reasoning and image caption. on all these tasks, unimo-2 obtains obvious improvements compared to the baselines that only learn on aligned image-caption data or without our grounded learning component. moreover, we also evaluate our model on single-modal textual tasks such as natural language inference and visual tasks such as image classification @xcite . the results show that our model has also achieved very impressive performance on these tasks, which proves the strong scalability and adaptability of our model. unimo-2 has the following advantages compared with previous methods: vision-language pre-training recent years have witnessed rapid progress in vision-andlanguage pretraining (vlp) @xcite @xcite @xcite . most existing mainstream vlp models adopt a two-stage training method, which firstly extracts region-based visual features using a pre-trained object detection model, and then combines the derived object-centric region features of images and text embeddings as the input of transformer @xcite for crossmodal pre-training. these methods rely heavily on an off-the-shelf object detector like faster r-cnn @xcite typically pretrained on the visual genome dataset @xcite . as the visual representation is not optimized towards a more generic cross-modal understanding and extracting region features with an object detection model is so time-consuming, they face the problems of limited visual expressive power and computation inefficiency, which makes them less scalable. some recent work has also explored vlp without object detection modules @xcite @xcite . they either utilize grid features from pretrained cnns or patch features following vit @xcite , however they only use limited imagecaption pairs for cross-modal pretraining and thus their scalability and performance are limited. only a few works have explored utilizing unaligned images and texts for vision-language pre-training, including our previous work unimo @xcite and u-visualbert @xcite . however, they all rely on pre-extraction of region-based visual features or object tags by time-consuming object detection. how to bridge unpaired visual and textual corpora end-to-end without using object detection remains challenging. text transformer [cls] a dog is running through the water [cls] grounded dictionary 1 2 3 4 5 c … grounded transformer grounded tokens image-text matching masked lm [cls] visual tokens text tokens grounded transformer grounded tokens visual contrastive learning visual transformer [cls] visual tokens text transformer text tokens grounded transformer grounded tokens masked lm [cls] text tokens grounded learning on image-text pairs grounded learning on unpaired images and text grounded dictionary 1 2 3 4 5 c … image patches text tokens image patches [cls] any baseball game involves one umpire standing behind the catcher… shows the architecture of learning on unpaired images and texts, which produces grounded tokens from image representations or text representations, respectively. as they share the same grounded dictionary, the grounded tokens act as "anchor points" to bridge the gap between images and texts. grounded learning language grounding is an active field aiming at enriching textual representations with visual information, which has been shown to improve performance on a variety of core nlp tasks @xcite @xcite . @xcite investigate grounded sentence representations by training a sentence encoder to predict the image features of a given caption. @xcite propose a vokenization method that maps language tokens to their related images. these works all enrich the language representation with visual information by learning a projection of text representations to corresponding images @xcite . recently, @xcite propose an end-to-end vlp method that aggregates visual features from a cnn encoder into visual tokens with a visual dictionary. @xcite propose to improve cross-modal retrieval tasks by incorporating a shared discretized embedding space, which is utilized to compute matching scores between different modalities to complement the representations from individual encoders. these works all rely on image-text pairs to learn cross-modal representations and only focus on joint vision-language tasks. by contrast, our work for the first time proposes to jointly model both aligned and unaligned images and texts by end-to-end learning a shared grounded semantic space, which can improve modality alignment between both aligned and unaligned images and texts. ## approach the overall architecture of our model is shown in figure 1 . unimo-2 is an end-to-end framework, which consists of a trainable transformer-based visual encoder, a transformer-based text encoder, a grounded dictionary (gd) embedding module, and a multi-layer grounded transformer for modality fusion. the visual encoder takes an image as input by splitting it into small sizes of patches, and produces the high-level visual representations for all patches, similar to vit @xcite . the text encoder encodes textual tokens to produce high-level token representations. based on the high-level representations of patches and tokens, we design a gd embedding module to group similar vision-language representations into grounded tokens with a shared grounded dictionary. the grounded transformer is further adopted to fuse features from vision and language modalities through interacting with the common grounded tokens. unimo-2 can be end-to-end pre-trained by joint masked language modeling (mlm) on text, image-text matching (itm) on image-text pairs and visual contrastive learning (vcl) on images. unimo-2 can also be easily adapted to various tasks including visual, textual and crossmodal tasks. ## experimental settings pretraining dataset our pre-training datasets consist of three types: text corpus, image corpus and image-text pairs. the text corpus includes two large-scale corpora: bookwiki and openwebtext, which are part of the training dataset of roberta @xcite . the image corpus are images without textual descriptions, including a subset of openimages @xcite and imagenet-21k @xcite . each image in these datasets contains a textual label. the image-text pairs are composed of four existing multi-modal datasets: coco @xcite , visual genome (vg) @xcite , conceptual captions (cc) @xcite and sbu captions @xcite , which have also been widely used in previous vlp models. the detail statistics are shown in the appendix. we also transform the label of each image to a sentence by prompts (e.g. "a photo of [label]") to create pseudo imagetext pairs from the openimages and imagenet-21k datasets for pretraining. implementation detail unimo-2 consists of 12 layers of visual transformer, 12 layers of text transformer, and 12 layers of grounded transformer. the visual transformer is initialized by vit-b/16. the text transformer and grounded transformer are both initialized by roberta-base. the maximum sequence length of text tokens are set as 512. an adam optimizer with initial learning rate 5e-5 and a learning rate linear decay schedule is utilized. for the visual encoder, our model receives the raw image @xmath0, where d is the fixed hidden size of the transformer layers and p is the patch size. during pretraining, we utilize the 224 × 224 resolution with a fixed patch size of 16 × 16, resulting in a patch sequence of length 14 × 14 as visual tokens. during fine-tuning, we increase the image resolution to 384 × 384 and interpolate the positional encoding of image patches following @xcite . for the grounded embedding module, the grounded dictionary size c is set as 2048, and the number of grounded tokens k during pre-training and finetuning are both set as 100 that is much smaller than the max number of patches and tokens for pre-training (i.e. 709) and finetuning (i.e. 1089). we set β = 0.25 in all our experiments and the results did not vary obviously for values ranging from 0.1 to 1.0. we have compared different grounding settings in detail in appendix a. finetuning tasks to show the scalability of our model, we fine-tune it on three types of downstream tasks: (1) joint vision-language cross-modal tasks, (2) visual tasks, and (3) textual tasks. the cross-modal tasks include: visual question answering (vqa) on the vqa v2.0 dataset @xcite , image caption on the microsoft coco captions dataset @xcite , visual entailment on the snli-ve dataset @xcite and image-text retrieval on flickr30k datasets @xcite . the visual tasks include image classification on the imagenet-1k dataset @xcite . the textual tasks include sentiment classification on the sst-2 dataset @xcite , natural language inference on the mnli dataset @xcite , linguistic acceptability analysis on the cola dataset @xcite and semantic similarity analysis on the sts-b dataset @xcite . the detail statistics of the datasets and hyper-parameter settings for the above tasks are described in appendix b. ## results and analysis we compare unimo-2 to a variety of state-of-theart models on cross-modal, visual and textual tasks. ## conclusion in this work, we propose unimo-2, an end-toend unified-modal pre-training framework that can learn from both aligned and unaligned image and text corpora. our proposed grounded learning method can help bridge unpaired images and texts and align the textual and visual semantic spaces more effectively. benefiting from effectively utilizing different types of corpora, unimo-2 has better scalability for different types of tasks. experiments show that unimo-2 greatly improves the performance of various cross-modal tasks and also achieves very impressive performance on visual and textual tasks. the results also show that it is promising to further uniformly improve the performance of cross-modal, visual and textual tasks by utilizing larger scales of unpaired images and texts.