M anager T ower: Aggregating the Insights of Uni-Modal Experts for Vision-Language Representation Learning (2023)

Abstract: Two-Tower Vision-Language (VL) models have shown promising improvements on various downstream VL tasks. Although the most advanced work improves performance by building bridges between encoders, it suffers from ineffective layer-by-layer utilization of uni-modal representations and cannot flexibly exploit different levels of uni-modal semantic knowledge. In this work, we propose ManagerTower, a novel VL model architecture that gathers and combines the insights of pre-trained uni-modal experts at different levels. The managers introduced in each cross-modal layer can adaptively aggregate uni-modal semantic knowledge to facilitate more comprehensive cross-modal alignment and fusion. ManagerTower outperforms previous strong baselines both with and without Vision-Language Pre-training (VLP). With only 4M VLP data, ManagerTower achieves superior performances on various downstream VL tasks, especially 79.15% accuracy on VQAv2 Test-Std, 86.56% IR@1 and 95.64% TR@1 on Flickr30K. Code and checkpoints are available at https://github.com/LooperXX/ManagerTower .

## introduction in recent years, there has been a growing interest in the field of vision-language (vl) representation learning due to the development of vision-language pre-training (vlp) techniques. vlp aims to learn transferable multi-modal knowledge from large-scale image-text pairs, which can further improve the performance of various downstream vl tasks, such as visual question answering @xcite , visual entailment @xcite , visual reasoning @xcite , and image-text retrieval @xcite . visual and textual modalities in vl models are typically processed by uni-modal encoders and subsequently fused in a cross-modal encoder. this general architecture can be referred to as the two-tower architecture. meter @xcite and bridgetower @xcite are two representative two-tower vl models. meter uses clip-vit @xcite and roberta @xcite as pre-trained uni-modal encoders, but it ignores different levels of uni-modal semantic knowledge in them and only feeds the last-layer outputs of each uni-modal encoder into the cross-modal encoder. in an effort to address this issue, as illustrated in figure 1(a) , bridgetower connects multiple top uni-modal layers with each cross-modal layer in a layer-by-layer fashion to exploit unimodal semantic knowledge at different levels. in this work, we build upon the research of bridgetower and advance it in two aspects. specifically, we address the limitations of bridgetower: (i) its layer-by-layer utilization of different unimodal layer representations is ineffective. each cross-modal layer can only utilize an artificiallyconnected uni-modal layer representation, thus restricting the exploitation of different levels of unimodal semantic knowledge. (ii) the number of cross-modal layers is tied to the number of uni-modal layer representations it used, thus limiting its scalability and capability. for example, increasing the number of uni-modal layer representations used requires a corresponding increase in the number of cross-modal layers. this leads to an increase in the number of parameters and computation cost, while does not always result in performance improvements as demonstrated by @xcite . as shown in figure 1(b) , we propose a novel vl model architecture, managertower, that aggregates multi-layer uni-modal representations via managers in each cross-modal layer. each manager takes multi-layer uni-modal representations as the insights of pre-trained uni-modal experts at different levels, and then adaptively aggregates them to facilitate more comprehensive cross-modal alignment and fusion. more concretely, inspired by the linear combination of layers @xcite ) method, we adapt it as the static aggregation of experts (sae) manager and then remove redundant information to design the static aggregation of uni-modal experts (saue) manager, which focuses on aggregating uni-modal semantic knowledge. we further propose the adaptive aggregation of uni-modal experts (aaue) manager to adaptively aggregate multi-layer uni-modal representations for each token in different cross-modal layers. moreover, in principle, managers can be easily integrated into any cross-modal encoders and work well with any uni-modal encoders, making managertower scalable and flexible. we first explore the feasibility of various designs of managers by evaluating and analyzing the performance on vqav2 and flickr30k datasets. then, we pre-train managertower with commonly used 4m vlp data and evaluate it on various downstream vl tasks. with the same pre-training and fine-tuning settings and uni-modal backbones as previous strong baselines such as meter and bridgetower, managertower achieves superior performances on various downstream vl tasks, especially 79.15% accuracy on vqav2 test-std, 86.56% ir@1 and 95.64% tr@1 on flickr30k. it outperforms not only many base-size models pre-trained on 4m data but also some models pretrained on more data and/or with larger size. ## preliminary in this work, for a fair comparison with meter and bridgetower, we use the same cross-modal encoder and pre-trained uni-modal encoders. ## manager design figure 2 depicts the overall framework of manager-tower. it introduces managers in each cross-modal layer to adaptively aggregate the insights of pretrained uni-modal experts at different levels. in the subsequent subsections, we will elaborate on the detailed design schema for the three types of managers, and conclude with the cross-modal encoder with our well-designed managers. foot_0 ## conclusion we propose managertower, a novel vl model architecture that gathers and combines the insights of pre-trained uni-modal experts at different levels via the introduced managers in each cross-modal layer. the feasibility of various designs of managers is well explored, and the effectiveness of manager-tower on various downstream vl tasks is well demonstrated. more comprehensive cross-modal alignment and fusion in each cross-modal layer is achieved by adaptive aggregation of different levels of uni-modal semantic knowledge. we hope that our work can inspire more research on how to better exploit multi-layer pre-trained uni-modal representations for cross-modal representation learning.