m PLUG : Effective and Efficient Vision-Language Learning by Cross-modal Skip-connections (2022)

Abstract: Large-scale pre-trained foundation models have been an emerging paradigm for building artificial intelligence (AI) systems, which can be quickly adapted to a wide range of downstream tasks. This paper presents mPLUG, a new vision-language foundation model for both cross-modal understanding and generation. Most existing pre-trained models suffer from inefficiency and linguistic signal overwhelmed by long visual sequences in cross-modal alignment. To address both problems, mPLUG introduces an effective and efficient vision-language architecture with novel cross-modal skip-connections.mPLUG is pre-trained end-to-end on large-scale image-text pairs with both discriminative and generative objectives. It achieves state-of-the-art results on a wide range of vision-language downstream tasks, including image captioning, image-text retrieval, visual grounding and visual question answering. mPLUG also demonstrates strong zero-shot transferability on vision-language and video-language tasks. The code and pre-trained models are available at https://github.com/alibaba/AliceMind

## introduction large-scale pre-training of vision-language models have recently received tremendous success on a wide range of cross-modal tasks @xcite @xcite @xcite @xcite . such vision-language models learn cross-modal representations from a quantity of image-text pairs by aligning the visual and linguistic modalities. the key to learning visionlanguage models is finding a good alignment between the two modalities to close the semantic gap in-between. most recent work @xcite @xcite @xcite proposes a direct alignment between the image and text representations in an endto-end manner, which turns out to perform well on many vision-language tasks. these models extract finer-grained visual representation with a long sequence of image patches or grids for good vision understanding @xcite . however, there are two significant problems in fusing the asymmetric information caused by long visual sequences: 1) vanishing information: the caption text in widely-used image-text pre-training data is usually short and highly abstract while more detailed and diverse information can be extracted from images as long visual sequences. this infor-mation asymmetry leads to linguistic signal overwhelmed by visual signal when information of the two modalities are directly blended. the linguistic signal can be overlooked, which hinders effective cross-modal fusion, and 2) inefficiency: cross-modal fusion focuses so hard on interaction between modalities that heavy and redundant selfattention on long visual sequences takes most computation time. this makes the cross-modal fusion rather inefficient. one straightforward way of cross-modal fusion is the connected-attention network as shown in @xcite . it adopts a single transformer @xcite network for early fusion of vision and language by simply taking the concatenation of visual and linguistic features as input @xcite . this paradigm allows self-attention to discover alignments between the modalities from the bottom level, and requires full self-attention on the concatenation of cross-modal sequences, which is rather time-consuming. besides, this type of methods process information from both modalities equally, which may suffer from the information asymmetry especially when there is a big difference in information density or sequence lengths between the modalities. another line of work employs separate transformer networks for textual and visual features, and uses techniques such as cross-attention to enable cross-modal interaction @xcite , as shown in figure 1 @xcite . this architecture design conducts cross-modal fusion on both modalities independently, which helps alleviate the information vanishing. however, it still suffers from computational inefficiency for full self-attention on long visual sequences, and it is not that parameter-efficient with two separate transformer networks. in this work, we propose mplug, a unified multi-modal pre-training framework for both vision-language understanding and generation. mplug performs effective and efficient visionlanguage learning with novel cross-modal skipconnections to address the problem of linguistic signal overwhelmed by visual signal. instead of fusing visual and linguistic representations at the same levels, the cross-modal skip-connections enables the fusion to occur at disparate levels in the abstraction hierarchy across the modalities. it creates inter-layer shortcuts that skip a certain number of layers for visual representations. in one way, it can skip heavy and redundant computation of self-attention between uni-modal visual tokens for efficiency. besides, we adopt the asymmetric crossattention from vision to language in certain layers so as to enhance the linguistic representation learning and alleviate information vanishing. as shown in figure 1 (c), in each block of our cross-modal skip-connected network, mplug first adopts an asymmetric co-attention architecture at the first few layers for efficiency, by removing the co-attention on vision which is timeconsuming due to long visual sequences. compared with the connected-attention network, this fusion method keeps more linguistic signal from being overwhelmed by visual signal. it is then followed by one layer of connected-attention, by concatenating the original visual representation and the co-attention output on the language side as input. this prevents the fused representation from being biased towards linguistic signal and forgetting visual signal. figure 1 shows that the new cross-modal skip-connected network achieves superior performance with at least four times speed-up than other cross-modal fusion networks. mplug pushes the state of the art on a wide range of visionlanguage tasks, including image captioning, imagetext retrieval, visual grounding and visual question answering. mplug also demonstrates strong zero-shot transferability on a wide range of visionlanguage and video-language tasks. ## conclusion this paper presents mplug, an effective and efficient vlp framework for both cross-modal understanding and generation. mplug introduces a new asymmetric vision-language architecture with novel cross-modal skip-connections, to address two fundamental problems of information asymmetry and computation efficiency in cross-modal alignment. pretrained on large-scale image-text pairs, mplug achieves state-of-the-art performance on a wide range of vision-language tasks. mplug also demonstrates strong zero-shot transfer ability when directly applied to multiple video-language tasks.