Deeply Coupled Cross-Modal Prompt Learning (2023)

Abstract: Recent advancements in multimodal foundation models (e.g., CLIP) have excelled in zero-shot generalization. Prompt tuning involved in the knowledge transfer from foundation models to downstream tasks has gained significant attention recently. Existing prompt-tuning methods in cross-modal learning, however, either solely focus on language branch, or learn vision-language interaction in a shallow mechanism. In this context, we propose a Deeply coupled Cross-modal Prompt learning (DCP) method based on CLIP. DCP flexibly accommodates the interplay between vision and language with a Cross-Modal Prompt Attention (CMPA) mechanism, which enables the mutual exchange of respective representation through a well-connected multi-head attention progressively and strongly. We then conduct comprehensive few-shot learning experiments on 11 image classification datasets and analyze the robustness to domain shift as well. Thorough experimental analysis evidently demonstrates the superb few-shot generalization and compelling domain adaption capacity of a well-executed DCP.

## introduction large foundation models pre-trained on web-scale image-text pairs such as clip @xcite and align @xcite have shown promising performance on zero-shot image classification. research has repeatedly shown that the general knowledge learned by the foundation models can also be transferred to diverse downstream tasks, such as few-shot image classification @xcite , visual grounding @xcite , visual question answering @xcite and so on. they have exhibited a significant potential in open-vocabulary scenarios. thus, the + work was done during internship at sensetime research * corresponding author challenge associated with how to efficiently and effectively adapt large pre-trained models to downstream tasks has garnered increasing attention especially in low-resource training scenarios. directly fine-tuning the foundation model is infeasible due to the massive training parameters and the catastrophic forgetting caused by overfitting @xcite . in contrast, the parameter-efficient prompt tuning approach explored in natural language processing has yielded significant success @xcite , leading to an increased examination of this technique within the realm of multi-modality, especially in the language-branch of clip. for example, coop @xcite and proda @xcite explore the vanilla few-shot learning based on clip by adjusting the embedding or distribution of the text prompt. cocoop @xcite and prograd @xcite focus more on the unseen classes. they contextualize the text prompt either under the supervision of visual clues or tweak gradient direction to improve the generalization ability of the model. the aforementioned approaches, however, only adjust the text embedding of clip and neglect the visual branch. the success of vpt @xcite demonstrates the effectiveness of visual prompt learning. inspired by this work, upt @xcite and @xcite synergize the visual and textual prompts. specifically, upt improves the few-shot learning ability by generating visual and text prompts initially. maple achieves better performance in the classification of unseen classes. they uncover the underlying rationale and limitations of dual-branch prompt tuning. concretely, the dual-branch clip learns the visual and language synergy only based on contrastive learning, whereas both branches lack mutual communication at the early stage of the network. multi-modal prompt learning techniques, such as maple and upt, incorporate languagevision interactions of the network and achieve substantially improved performance, highlighting the significance of the cross-modal interactions. however, previous studies have leveraged languagevision interactions at a superficial level. for example, upt generates visual and text prompts before they are fed into the corresponding encoders. maple generates visual prompts conditioned on language counterparts by a mapping function. many studies @xcite have shown that neural networks, especially transformer-based models, can leverage the deep fusion of information from multiple views to improve their performance. it remains less explored in the thread of multi-modal few-shot learning. to this end, we design deeply coupled cross-modal prompt learning (dcp) enhancing the language-vision interaction. specifically, dcp is built upon clip, with additional text and visual prompts across multiple layers. different from previous methods with deep prompt tuning @xcite @xcite , dcp only initializes the first layer of visual and text prompt randomly. the subsequent prompts are generated by cross-modal prompt attention (cmpa) module, which elegantly integrates the prompts from the preceding cross-modal layer. cmpa is characterized with stronger connection in two folds, i.e., depth and breadth. 1) depth means that cmpa intensifies the correlation of the prompts among different layers. 2) breadth refers to that cmpa amplifies the interaction between visual and language modalities. cmpa is the core module to realize the deep coupling between two modalities. essentially, dcp empowered by cmpa amalgamates uni-branch and dual-branch multi-modal pre-training paradigms in a favorable way in an attempt to bridge the discrepancy between visual and textual knowledge without introducing too much overhead. to conclude, the contributions of this work are as follows: our method surpasses visual prompt tuning, text prompt tuning and existing competitive multi-modal prompt tuning methods under the few-shot setting. ## method in this section, we first introduce the preliminaries, including clip @xcite , coop @xcite and vpt @xcite . then, we describe our deeply coupled prompt learning (dcp) and detail its underlying module cmpa. ## experiments in this section, we conduct experiments to evaluate the effectiveness of our method under two settings. one is few-shot visual recognition including 11 different datasets covering generic objects, scenes, actions and fine-grained categories. the other is domain adaptation, where we train our model on imagenet and evaluate it on other four datasets. ## discussion and conclusion this paper proposes a deeply coupled cross-modal prompt learning method, with a core module crossmodal prompt attention. our method focuses on optimizing the interaction across different models and layers to address the alignment between vision and language. experiments on few-shot image clas-sification and domain adaptation evidence that our method can transfer the general knowledge learned by pre-trained foundation models to downstream tasks without penalty of the original generalization ability. our method provides a strong baseline on few-shot image classification. the deep fusion between visual and language information may enable our approach to have greater potential for complex cross-modal tasks, such as referring expression comprehension @xcite , image retrieval @xcite and visual question answering @xcite . we will apply our method to such complicated cross-modal tasks to evaluate its effectiveness in our future work.