X treme CLIP : Extremely Parameter-efficient Tuning for Low-resource Vision Language Understanding (2023)

Abstract: Recently, Contrastive Visual-Language Pre-training (CLIP) has demonstrated remarkable capability in various Visual Language Understanding (VLU) tasks. Yet, most CLIP-based methods require tasks-specific designs and sufficient training data. In this paper, we introduce a simple yet efficient paradigm for low-resource VLU named XtremeCLIP, which involves very few trainable parameters to improve the generalization ability of the trained models. In our XtremeCLIP framework, we reformulate a series of VLU tasks as a unified open-book affinity-matching problem. Furthermore, to handle the insufficient supervised signals in small datasets, we adopt contrastive learning to utilize the implicit sorting information of ground-truth labels to provide more supervised cues. Extensive experiments over multiple datasets on visual entailment, visual question answering, and image classification show that XtremeCLIP consistently outperforms existing baselines in low-resource settings.

## introduction pre-trained visual-language models such as x-vlm @xcite and clip @xcite have been proposed to unify visual and textual representations in the same embedding space and shown great potential for visual language understanding (vlu). conventional fine-tuning approaches @xcite @xcite heavily depend on the time-consuming and labor-intensive process of data annotation, which are bothersome in low-resource scenarios. in the literature, ben @xcite ; @xcite propose partial-parameter fine-tuning to preserve the pre-trained knowledge of these models. @xcite ; @xcite ; @xcite reformulate visual grounding and visual question answering as a "fill-inblank" problem by hand-crafted prompts. @xcite ; @xcite utilize lightweight adapters @xcite to retain the knowledge of clip. @xcite ; @xcite address image classification tasks by utilizing textual representations describing image categories. despite the success, we suggest there are still some drawbacks in existing works. i) the discrete prompt paradigm requires labor-intensive promptengineering, while the soft template paradigm results in an unstable training process. ii) adapters or partial-parameter fine-tuning methods may underperform due to their relatively large number of tunable parameters, requiring additional training data to achieve satisfactory results. iii) the aforementioned methods are task-specific in design, implying that their effectiveness may be derived from task-specific architectures. hence, it is vital for us to design a more unified parameter-efficient tuning approach in order to solve various vlu tasks. in this paper, we present xtremeclip, an extremely parameter-efficient tuning method for solving various vlu tasks based on clip @xcite . xtremeclip reformulates a series of vlu tasks uniformly into an open-book affinitymatching problem. here, we adopt a knowledgebase prototype matrix to record the salient characteristics for each class by visual-textual fusion features, then perform affinity matching between image-text pairs and prototypes of each class. we further utilize the implicit sorting information of ground-truth labels by contrastive learning to provide more supervised cues from low-resource training sets. during model training, all parameters of textual and visual encoders in clip are fixed. hence, xtremeclip is extremely parameterefficient. we conduct extensive experiments on a visual entailment (ve) benchmark (i.e., snli-ve), a visual question answering (vqa) benchmark (i.e., vqa v2), and three widely used image classification (ic) benchmarks (i.e., eurosat, dtd, and fgvc). results show xtremeclip consistently outperforms baselines in low-resource scenarios. the model architecture and training procedure of xtremeclip are in figure 1 . first, a knowledgebase prototype matrix is constructed @xcite by combining visual and textual features, designed to serve as a repository of the key characteristics for each class. then, open-book affinity matching is performed between the image-text instance and the prototypes for each class. ## conclusion we propose xtremeclip, a simple and efficient paradigm that reformulates vlu tasks as a prototype affinity matching problem. we adopt contrastive learning to leverage implicit sorting information from ground-truth labels, providing more supervised cues to handle insufficient supervised signals in small datasets. experimental results demonstrate that xtremeclip consistently outperforms all baselines in low-resource scenarios. ming zhong, pengfei liu, yiran chen, danqing wang, xipeng qiu, and xuanjing huang. 2020. extractive summarization as text matching. in acl, pages 6197-6208. kaiyang zhou, jingkang yang, chen change loy, and ziwei liu. 2022a. conditional prompt learning for vision-language models. in cvpr, pages 16816-16825. kaiyang zhou, jingkang yang, chen change loy, and ziwei liu. 2022b. learning to prompt for vision-language models. ijcv, pages 2337-2348. beier zhu, yulei niu, yucheng han, yuehua wu, and hanwang zhang. 2022. prompt-aligned gradient for prompt tuning. arxiv.