E 2 E - VLP : End-to-End Vision-Language Pre-training Enhanced by Visual Learning (2021)

Abstract: Vision-language pre-training (VLP) on large-scale image-text pairs has achieved huge success for the cross-modal downstream tasks. The most existing pre-training methods mainly adopt a two-step training procedure, which firstly employs a pre-trained object detector to extract region-based visual features, then concatenates the image representation and text embedding as the input of Transformer to train. However, these methods face problems of using task-specific visual representation of the specific object detector for generic cross-modal understanding, and the computation inefficiency of two-stage pipeline. In this paper, we propose the first end-to-end vision-language pre-trained model for both V+L understanding and generation, namely E2E-VLP, where we build a unified Transformer framework to jointly learn visual representation, and semantic alignments between image and text. We incorporate the tasks of object detection and image captioning into pre-training with a unified Transformer encoder-decoder architecture for enhancing visual learning. An extensive set of experiments have been conducted on well-established vision-language downstream tasks to demonstrate the effectiveness of this novel VLP paradigm.

## introduction self-supervised pre-training has achieved great success in a wide range of natural language understanding @xcite @xcite and generation tasks @xcite @xcite . recent studies @xcite @xcite @xcite have also witnessed the progress of self-supervised pretraining on vision-and-language tasks, which learns * corresponding author general cross-modal representations from massive image-text pairs, and fine-tunes vision-language pre-training (vlp) models on task-specific data achieving state-of-the-art results on various downstream v+l tasks. most existing mainstream vlp models adopt a two-step training method, which firstly extracts semantic visual features using a pre-trained object detection model, and then combines the derived object-centric representation of the image and text embedding as the input of transformer @xcite for cross-modal pre-training. despite the superior performance brought by the large-scale image-text pairs, the two-stage solution suffers from the following weaknesses: 1) the object detection model in the first step is trained on specific visual dataset such as visual genome dataset @xcite , and the visual representation is not optimized towards a more generic cross-modal understanding in the second step. it may suffer from an error propagation problem when the object detection model fails to recognize certain important information. 2) extracting region features with an object detection model is so time-consuming that most state-of-the-art models are directly trained and evaluated on cached visual features. this practice not only imposes unnecessary constraints on model designs, but also confronts the run-time inference inefficiency in the prediction phase. recently, several studies such as @xcite have begun to revisit the grid features for cross-modal understanding and found the grid features can also work surprisingly well, while making the model design and training process much simpler. one pioneering work pixel-bert @xcite explores to pre-train with grid features in an end-to-end fashion directly from pixels. it removes all the fine-grained visual pre-training tasks, which proves to be important for v+l pre-training. @xcite also demonstrates that visual features provided by the object detection model matter significantly in vlp models. to address the limitations, we propose a new endto-end paradigm for pixel-level vision-language pre-training, namely e2e-vlp, by enhancing with fine-grained visual learning. during pre-training, e2e-vlp jointly learns the visual region features and the cross-modal representation in a unified transformer encoder-decoder architecture directly from image pixels. in addition to the typical pre-training tasks of masked language modeling and image-text matching, we enhance the visionlanguage pre-training with fine-grained visual semantic learning. specifically, two end-to-end pretraining tasks are further incorporated: 1) object detection: inspired from detr @xcite , we view the object detection as a direct set prediction problem. the cross-modal transformer encoder and image encoder are joint learnt to fuse the cross-modal data from pixels, while the decoder is used to capture fine-grained visual information via bipartite matching between predicted and ground-truth objects; 2) image-text generation: to better understand the semantics within the image, we also use the paired text to guide the learning of image features. we use the encoder network to represent the image and a left-to-right decoder to generate the caption text. the standard auto-regressive language model objective is used to maximize the data probability. these two tasks can help learn high-quality visual representations @xcite . detection task can learn object-level visual semantics, while the image caption task can capture textaligned visual semantics. these two kinds of visual semantics matter significantly in vlp cross-modal fusion. during fine-tuning, e2e-vlp can be flexibly applied to vision-language understanding tasks with the encoder module, and vision-language generation tasks with the encoder-decoder module. we evaluate e2e-vlp on a variety of representative vision-language tasks, including visual question answering, natural language visual reasoning, cross-modal retrieval and image captioning. with the new end-to-end pre-training paradigm, we can obtain surprising good performance across different v+l tasks and greatly decrease the online inference time with the new one-stage solution. we make the following major contributions in this paper: ## related work self-supervised pre-training has substantially advanced the performance across a variety of natural language understanding @xcite @xcite ) and text generation tasks @xcite @xcite . inspired by language model pre-training, several researchers propose visionlanguage pre-training(vlp) models on large-scale image-text pairs, which has proved effective for a wide range of vision-language (vl) tasks, such as vqa @xcite , nlvr @xcite , cross-modal retrieval @xcite . the current vlp models mainly take two-step training pipeline, which consists of extracting semantic visual features by object detector and training the cross-modal pre-training model to align text and visual features. in this kind of method, there are mainly two broad directions to conduct visionlanguage pre-training. the first line uses a singlestream transformer architecture @xcite to model both image and text representations in a unified semantic space such as vlbert @xcite , uniter @xcite and oscar @xcite . in contrast, the other line adopts a two-stream transformer architecture that first encodes the image and text modalities separately, and then fuses the cross-modal representations with another transformer network, such as lxmert @xcite and ernie-vil @xcite . besides, semvlp @xcite is pre-trained iteratively with two prevalent fashions. these methods are directly trained and evaluated on cached visual features, which imposes unnecessary constraints on model designs and makes it hard to enable an end-to-end visionlanguage pre-training. furthermore, pixel-bert figure 1 : the overall framework of e2e-vlp. our model employs a unified encoder-decoder transformer framework to learn visual representation, and semantic alignment between image and text jointly. @xcite represents the first and only work to pre-train with grid features in an end-toend fashion. however, due to the characteristics of learnt grid features, the end-to-end pre-training is conducted without object-level visual tasks, which is important in aligning the semantics between cross-modal representations. in this paper, we focus on enhancing the end-toend vision-language pre-training with more finegrained visual semantic learning. the object detection task and image caption task are incorporated into the pre-training stage for further improving the fine-grained visual-language understanding and generation abilities. ## conclusion in this paper, we propose a new end-to-end paradigm for pixel-level vision-language pretraining, to jointly learn visual representation, and semantic alignments between image and text. different from the previous methods using the region features in a two-stage pipeline, we propose to use the more flexible and efficient image grid features for vision-language pre-training. we further incorporate the tasks of object detection and image captioning into pre-training with a unified transformer encoder-decoder architecture for enhancing visual learning. the experiments on well-established vision-language downstream tasks demonstrate the effectiveness and efficiency of our e2e-vlp model. we hope that this study can potentially offer new perspectives and guide for endto-end vision-language pre-training. in the future, we will explore more deeply interacted ways for image-text fusion from a bottom layer, and incorporate more advanced vision and language pre-training tasks for further improving the performance.