Toward Building General Foundation Models for Language, Vision, and Vision-Language Understanding Tasks (2023)

Abstract: Foundation models or pre-trained models have substantially improved the performance of various language, vision, and vision-language understanding tasks. However, existing foundation models can only perform the best in one type of tasks, namely language, vision, or vision-language. It is still an open question whether it is possible to construct a general foundation model performing the best for all the understanding tasks. In this paper, we propose a new method for training the general foundation model, X-FM (the X-Foundation Model). X-FM has one language encoder, one vision encoder, and one fusion encoder, as well as a new training method. The training method includes two new techniques for learning X-FM from text, image, and image-text pair data. One is to stop gradients from the vision-language training when learning the language encoder. The other is to leverage the vision-language training to guide the learning of the vision encoder. Extensive experiments on benchmark datasets show that X-FM can significantly outperform existing general foundation models and perform better than or comparable to existing foundation models specifically for language, vision, or vision-language understanding. Code and pre-trained models are released at https://github.com/zhangxinsong-nlp/XFM.

## introduction with the enormous power of foundation models, also known as pre-trained models, remarkable performance gains have recently been achieved in a variety of understanding tasks in natural language processing (nlp), computer vision (cv), and other fields (devlin et al., 2019; liu et al., 2019; lewis et al., 2020; raffel et al., 2020; brown et al., 2020; dosovitskiy et al., 2021; he et al., 2022; bao et al., 2021; lu et al., 2019; tan and bansal, 2019; chen * correspondence to: <xszhang0320@gmail.com>. @xcite @xcite @xcite . foundation models are usually equipped with transformer @xcite as the backbone, pre-trained with a tremendous amount of unlabeled data, and then fine-tuned with small amounts of labeled data in downstream tasks. the strong representation ability of the model, the massive amount of data, and the effective means of training make the foundation models powerful for successfully solving the tasks of vision, language, and vision-language @xcite @xcite @xcite . the state-of-the-art foundation models usually work the best for one type of tasks, namely language, vision, and vision-language. for example, roberta @xcite , beitv2 @xcite , and x-vlm @xcite are language, vision, and vision-language foundation models respectively, and can achieve stateof-the-art performances for the specific type of tasks. it is still very challenging, however, to build a general foundation model that can perform the best in all types of tasks. existing models, such as flava @xcite , ofa @xcite , davinci @xcite and uni-perceiver-moe @xcite , are trying to achieve the goal. their performances are still not satisfactory, however, when compared with the best performing foundation models for the individual types of tasks, as shown in methods text tasks vision tasks multi-modal tasks (mscoco retriveal & vqa & nlvr) glue imagenet zero-shot fine-tune mnli ft/le tr ir tr ir vqa nlvr in learning of x-fm, the language encoder, vision encoder, and fusion encoder are jointly trained with text data, image data, and image-text pair data as input. given the text data, we train the language encoder by masked language modeling (mlm). given the image data, we train the vision encoder by masked image modeling (mim). given the image-text pair data, we train the fusion encoder by image text matching (itm), image-conditioned masked language modeling (imlm), bounding box prediction (bbp), also train the vision encoder and the language encoder by image-text contrastive learning (itc). (see fig 1 . @xcite the essential thinking of our learning method is that language is more abstract than vision, and there is an asymmetric relationship between language and vision. therefore, we separate the learning of the three encoders. the language encoder is trained mainly from text data and is isolated from the training of the fusion encoder. the vision encoder is simultaneously trained from image data and image-text pair data, guided by the visionlanguage training. the fusion encoder is trained from image-text pair data. our learning method includes two new techniques. one technique is to stop gradients from the vision-language training when learning the language encoder. the gradient flow is stopped from the fusion encoder to the language encoder in training, while the activation flow from the language encoder to the fusion encoder is as usual. as a result, the language encoder is not affected by training of the fusion encoder with image-text pair data. moreover, the training of the fusion encoder concentrates on learning the alignments between language and vision features. the other technique is to leverage the visionlanguage training to guide the learning of the vision encoder with masked image modeling (mim). in mim, the masked image is compared with the original image by the differences between the predicted representations and target representations at the masked and [cls] positions. the vision encoder creates both the predicated and target rep-resentations, while there is gradient flow from the predicted representations but no gradient flow from the target representations. the vision encoder can create the target representations because it is also trained in the vision-language training. we conduct experiments on a variety of twentythree tasks of language, vision, and vision-language understanding. x-fm can outperform other general foundation models by a large margin and can even achieve better or comparable performance than sota foundation models specifically designed for language, vision, or vision-language understanding tasks, as shown in our contribution is as follows. ## related work following the success of language model pretraining @xcite @xcite @xcite @xcite , vision pre-training @xcite @xcite @xcite and vision-language pretraining @xcite @xcite @xcite @xcite @xcite with transformer as the backbone have also made significant progress recently, pushing the state-ofthe-art of various understanding tasks of language, vision, and vision-language. recently, the fact that transformer can model multi-modal data within a single architecture has inspired research to develop general foundation models that can solve language, vision, and vision-language tasks at the same time. unimo @xcite jointly learns vision representations, language representations, and vision-language alignments in a shared space. flava @xcite performs pre-training with masked uni-modal and multi-modal modeling objectives. ofa @xcite formulates visionlanguage tasks as sequence-to-sequence (seq2seq) problems and pre-trains a seq2seq model in multitask learning. simvlm @xcite pretrains a seq2seq model with a single objective of language generation (prefix language modeling). davinci @xcite combines prefix language modeling and prefix image modeling to learn a general foundation model for a wide range of tasks. uni-perceiver @xcite builds a unified perception architecture that processes various modalities and tasks with a single transformer and shared parameters. previous studies on general foundation models have shown that different capabilities can be established with only one model. still, few studies demonstrate that the best performance can be achieved in all tasks with one model. in this paper, we propose a new method for training general foundation model and show that it can perform the best for all the understanding tasks of language, vision, and vision-language. we compare our model extensively with recent general foundation models on multiple dimensions, as shown in appendix a. several super-large foundation models (over 1b parameters) are proposed recently, most of which are trained on super-large in-house datasets (over 900m image-text pairs). the authors do not report results at the base (about 300m parameters) scale on public datasets, which we consider in this paper. coca @xcite pre-trains an image-text sequence-to-sequence model with contrastive loss and captioning loss. beit-3 @xcite ) uses a multi-way transformer and a unified objective of masked "language" modeling for learning from image, text, and image-text pair data. flamingo @xcite makes use of a large language model in vision-language pre-training to solve the "in-context learning" problem for vision-language tasks. pali @xcite jointly scales up the vision encoder and language encoder to cover a variety of language, vision, vision-language, and multilingual tasks. text encoder cross-attention fusion encoder feed forward self-attention vision encoder feed forward self-attention two brown and white dogs. given a text, we learn the language encoder by mlm. given an image, we learn the vision encoder by mim. given an image-text pair, we learn the fusion encoder by bbp, itm, imlm and itc, and further learn the vision encoder by mim. the gradients of bbp, itm, and imlm are stopped from the fusion encoder to the language encoder. the vision encoder is trained by mim with both the image-text pair data and the image data. m, n and l denote numbers of encoder layers. ## limitations and potential risks limitations. like most existing work on foundation models, the entire project consumed over 5 a100 gpu years on a computing cluster with high electricity costs, although we only tested base models. there is still potential for efficiency improvement through sparse attention @xcite or the lottery ticket hypothesis @xcite . we will explore the techniques to improve the training efficiency and reduce the carbon footprint so that we can adhere to the proposals on "green" deep learning @xcite . ## conclusion in this work, we address the problem of how to build a general foundation model that can perform the best for all the understanding tasks of language, vision, and vision-language. we propose a new method for training general foundation model with two new and effective techniques, bringing in x-fm, to learn rich language, vision, and visionlanguage representations at the same time. experimental results demonstrate that x-fm outperforms other general foundation models by a large margin. moreover, x-fm can even be better than or comparable to the sota foundation models specifically designed for language, vision, or vision-language understanding tasks.