Fusion or Defusion? Flexible Vision-and-Language Pre-Training (2023)

Abstract: Existing approaches in the vision-and-language pre-training (VLP) paradigm mainly deploy either fusion-based encoders or dual-encoders, failing to achieve both effectiveness and efficiency in downstream multimodal tasks. In this paper, we build a flexible VLP model by incorporating cross-modal fusions into a dual-encoder architecture, where the introduced fusion modules can be easily decoupled from the dual encoder so as to switch the model to a fusion-free one. To better absorb cross-modal features from the fusion modules, we design a cross-modal knowledge transfer strategy along with other comprehensive pre-training tasks to guide the training process, which can further strengthen both the fusion-based and fusion-free representation learning. Extensive experiments conducted on various downstream vision-language tasks show that our proposed model is well-equipped with effectiveness as well as efficiency, demonstrating a superior performance compared with other strong VLP models.

## introduction with the great development of self-supervised pretraining in both the community of natural language processing @xcite and computer vision @xcite , recent researches have also witnessed the success of vision-and-language pretraining (vlp). vlp learns generic multimodal representations from large-scale image-text pairs and can be further finetuned on various downstream vision-language (vl) tasks, including image-text retrieval @xcite , visual question answering @xcite , visual reasoning @xcite and visual entailment @xcite . the core of vlp resides in modeling the interaction between image and text representations. most of the mainstreams first represent the input image via pre-trained deep feature extractors, then feed the derived visual features along with the text embeddings into multi-layer transformers @xcite , in which cross-modal attention is used to fuse multimodal representations. despite demonstrating superior performances on downstream vl tasks, the fusion-based methods need to jointly encode image and text representations, significantly degrading the efficiency in retrieval tasks with massive candidates of image-text pairs. to make vlp models applicable in real-world scenarios, another line of methods independently encode text and image with dual encoders, shown in fig. 1(a ), in which cross-modal fusion is conducted by lightweight modules such as dot production. thanks to the dual-encoder architecture, encoded features of image and text can be precomputed offline for inference efficiency. nevertheless, independent encoding with shallow interaction fails to fully exploit the cross-modal interaction, making the performance far from satisfactory in vl classification tasks that require a strong ability of multimodal reasoning. a cute shiba inu in yellowish brown color is staring at you training image text "associated" images learning a s s o c ia te image candidates closer farther ✓ ✘ enhanced text features figure 2 : an example intuitively illustrates why a fusion-free text encoder can still work when the cross-modal fusions are removed from it. during training, the cross-modal fusions teach the text feature to "associate" what the related images could be. in inference of image-text retrieval, since a relevant image candidate is naturally closer to other images that are also related to the same text, the text feature along with its "associated" images will become closer to the relevant candidates than the original text feature. in this paper, we propose a unified and flexible vlp model named fod, which incorporates cross-modal fusions into a dual-encoder architecture for achieving both efficacy and efficiency in multimodal scenarios. specifically, we adopt a dual architecture with one image encoder and one text encoder, in which cross-modal fusions are placed in the text encoder side. considering that conventional fusions are based on either concatenation @xcite or cascading @xcite that can't be directly decoupled from the boarding encoder, we employ a parallel-style fusion module to model cross-modal interactions, shown in fig. 1 . in this way, fod can explicitly capture the complex interaction between modalities during training while switching the fusion-based text encoder to a fusionfree one by removing the fusion module. in order to retain more cross-modal knowledge in fod when the fusion modules are removed, we further design a cross-modal knowledge transfer strategy that forces both the unimodal features of image and text to approximate the multimodal representation produced by the fusion-based encoder. intuitively, since paired image and text describe the same object in different views, we can naturally associate a set of relevant images when given a caption (and vice versa). thus, if the text feature learns to "associate" its related images and absorbs them to enhance itself, the enhanced text feature can become closer to the relevant image candidates (and also farther to the unrelated ones) in inference. a concrete example illustrating this intuition is shown in fig. 2 . we evaluate our model on both image-text retrieval tasks and vision-language understanding tasks. experimental results show that our model outperforms other vlp methods on all downstream vl tasks, and even performs competitively with models that use a larger order of magnitude of data for pre-training. thanks to the detachable fusion module and the strategy of knowledge transfer, our model can be flexibly switched to a fusion-free pattern to enjoy a much faster inference speed of retrieval while retaining most of the performance. ## related work without considering the ways of visual feature extraction, the approaches of vision-language pretraining can be divided into two categories based on the interaction form between image and text. the first category, fusion-based model, explicitly utilizes deep fusion layers with cross-modal attention to model the interaction of images and texts @xcite @xcite @xcite 2020, 2021b; @xcite @xcite @xcite @xcite @xcite . these models perform well on vision-language understanding tasks due to the ability of capturing deep cross-modal features. however, for vision-language retrieval tasks, the fusion-based methods need to encode all the possible image-text pairs to find the most relevant candidate, resulting in extremely high time cost. the second category, dual-based model, utilizes a visual encoder and a text encoder to separately encode images and text, while the interaction between images and text is modeled by cosine similarity or linear projection @xcite @xcite . although dualbased models are effective for retrieval tasks since features can be pre-computed and cached offline, the shallow interaction is insufficient to tackle the vision-language understanding tasks that require complex vl reasoning. besides, training a dualbased model often necessitates a large number of image-text pairs (e.g. 300m for filip @xcite and 1.8 b for align @xcite ). recently, some researchers have devoted themselves to investigating a unified model that is wellperformed on vision-language understanding tasks while maintaining the efficiency towards retrieval tasks @xcite @xcite @xcite . to achieve this, one line of the works leverage knowledge distillation, in which a fusion-encoder model is pre-trained as a teacher model to guide the training of a dual-encoder model @xcite , but the performance is inevitably limited by the teacher model. other efforts attempt to train a modality-agnostic encoder with shared parameters, which can be used as either a fusion encoder or a dual encoder @xcite . despite the benefits of modeling all the modalities into a single encoder, it is hard to fully train such a huge model and a large number of training samples in different modalities are required. different from these methods, we incorporate a detachable cross-modal fusion module into a dualencoder architecture, which can easily remove the fusion module in inference and switch to a fusionfree model. more importantly, our model does not rely on teacher models or massive data in other modalities. ## model architecture as shown in fig. 3 , fod is in a transformer-based dual-encoder architecture that includes a visual encoder and a text encoder. the text encoder can be flexibly switched between a fusion-based pattern and a fusion-free pattern. for the fusion-based pattern, cross-modal fusions are incorporated into the text encoder to model multimodal interactions. for the fusion-free pattern, the fusion module is decoupled from the text encoder so as to get rid of the cross-modal calculation. during training, both fusion-based and fusion-free patterns are involved in the learning process, while in inference, the text encoder will be switched to one of the two patterns according to the type of downstream tasks. in the following sections, we introduce the visual encoder and the two patterns of the text encoder, followed by the pre-training strategies. ## qualitative analysis we further provide a qualitative analysis by using grad-cam @xcite to illustrate the per-word visualizations of the cross-modal attention maps of the fusion-based encoder. as shown in fig. 5 , from the visualizations we observe that when conducting image-text matching tasks, our model can focus on specific regions in an image according to different words in each sentence, including objects, actions, attributes and background. more examples are given in appendix. ## conclusion in this work, we propose a flexible vlp model that incorporates cross-modal fusions into a dualencoder architecture. the fusion module can be easily decoupled in inference, enabling the model to be switched between fusion-based and a fusionfree patterns according to different scenarios. extensive experiments conducted on both image-text retrieval and vision-language understanding tasks show that our model is well-equipped with effectiveness and efficiency compared with existing vlp models.