Vision-Language Model Fine-Tuning via Simple Parameter-Efficient Modification (2024)

Abstract: Recent advances in fine-tuning Vision-Language Models (VLMs) have witnessed the success of prompt tuning and adapter tuning, while the classic model fine-tuning on inherent parameters seems to be overlooked. It is believed that fine-tuning the parameters of VLMs with few-shot samples corrupts the pre-trained knowledge since fine-tuning the CLIP model even degrades performance. In this paper, we revisit this viewpoint, and propose a new perspective: fine-tuning the specific parameters instead of all will uncover the power of classic model fine-tuning on VLMs. Through our meticulous study, we propose ClipFit, a simple yet effective method to fine-tune CLIP without introducing any overhead of extra parameters. We demonstrate that by only fine-tuning the specific bias terms and normalization layers, ClipFit can improve the performance of zero-shot CLIP by 7.27% average harmonic mean accuracy. Lastly, to understand how fine-tuning in CLIPFit affects the pre-trained models, we conducted extensive experimental analyses w.r.t. changes in internal parameters and representations. We found that low-level text bias layers and the first layer normalization layer change much more than other layers. The code will be released.

## introduction large pre-trained visual-language models (vlms) have been developed a lot in recent years. for example, clip @xcite and alig @xcite demonstrated remarkable performance for various tasks, e.g., image recognition in a zero-shot fashion. to further improve the performance on the specific downstream tasks, prompt tuning @xcite @xcite and adapter tuning @xcite methods have been proposed. as shown in fig. 1 , prompt tuning methods proposed to introduce a set of learnable prompt vectors as the input of the text encoder while adapter tuning approaches adopted an additional bottleneck layer to learn new features. during the fine-tuning procedure, both of these two strategies keep clip's parameters fixed. the performance of prompt tuning and adapter tuning methods are superior on various tasks @xcite , so research on fine-tuning the inherent parameters of vlms has been barely touched. for language models, fully fine-tuning with downstream data can achieve promising results @xcite . moreover, recent works in language model fine-tuning (e.g., @xcite ) have demonstrated that, without introducing any external parameters, finetuning only the bias terms in a pre-trained model can perform competitively on downstream tasks compared with fine-tuning the entire model. for vlms, however, it is believed that fine-tuning the parameters of vlms corrupts the inherent pretrained knowledge as fully fine-tuning degrades performance @xcite . in this paper, we revisit this viewpoint and ask if, without introducing any external parameters, fine-tuning the inherent parameters of vlms can achieve competitive performance compared with prompt tuning. we start with directly applying bitfit to finetuning the clip model. we explore two strategies: (i) applying bitfit to the text encoder alone, and (ii) applying bitfit to both the text and image encoder. we found that both two strategies can acquire taskspecific knowledge but their performance to unseen class data can be poor (more discussed in sec. 4.4), implying that directly fine-tuning the bias terms of a text or image encoder may harm the model's generalization ability. these findings motivate us to develop more effective and efficient fine-tuning techniques for vlms. in light of this, we propose clipfit, a simple yet effective method for efficiently fine-tuning vlms. clipfit is orthogonal to previous prompt tuning and adapter tuning methods, as shown in fig. 1 (c ). for the text encoder, instead of fine-tuning all the bias terms, clipfit proposes to tune only the bias terms of projection linear layers in feed-forward networks (ffns). fine-tuning only the bias terms of projection linear layers in ffns will reduce the number of training parameters compared with finetuning all the bias terms. moreover, empirically, we discovered that our bias term tuning strategy can generalize better than bitfit @xcite , as shown in sec. 4.4. for the image encoder, as mentioned before, it may harm the model's performance if directly applying bitfit. in the image encoder, layer normalization (layernorm) @xcite aims to normalize the distributions of intermediate layers. since the distributions of pretraining and downstream data might be divergent, pre-trained layernorm might lead to sub-optimal performance for downstream data inference. therefore, clipfit proposes to further update only the parameters of the image encoder's layernorm. updating layernorm can yield a better image encoder for downstream data. lastly, previous studies @xcite have shown that generic pre-trained knowledge is easily forgotten in the fine-tuning stage. therefore, we explored two different regularization strategies for alleviating forgetting: (i) using the knowledge distillation (kd) loss @xcite to guide clipfit to learn from the zero-shot clip; (ii) using the mean squared error (mse) loss in bias terms to penalize changes in text encoder. we empirically found that both two strategies can alleviate forgetting problems and the kd loss performs better, thus we used the kd loss as the final solution for clipfit. fine-tuning is an empirical and black-box process. so, understanding how fine-tuning affects the pre-trained models is important for uncovering the black-box fine-tuning process. previous works @xcite @xcite explored this for language models fine-tuning. however, very little work explored the internal black-box fine-tuning process for vlms. in this paper, we conducted an initial exploration to analyze vlm fine-tuning process of clipfit, focusing on changes in internal parameters and representations. we found that for bias terms in the fnn of the text encoder, as the number of layers increases, the change in bias decreases, which means that during the fine-tuning process, low-level features in the text encoder change more than high-level features. for layernorm in the image encoder, we found that the first layer (patch embedding) changes much more than other layers. experimentally, we showed that more changed layers play a more important role in adapting downstream knowledge than less changed layers. moreover, we explored how kd loss affects the finetuning process for alleviating forgetting. we found that kd loss will reduce the changes for the morechanged low-level bias terms and enhance changes in less-changed high-level layers, which implies that penalizing changes for low-level bias terms is important for avoiding overfitting. lastly, we found that tuning layernorm will form a better image feature space compared with zero-shot clip. we conducted extensive experiments on 11 datasets in 4 different settings to show the effectiveness of the proposed clipfit. overall, our main contributions can be summarized as follows: ## methodolgy in this section, we introduce clipfit. we first briefly review clip and then illustrate clipfit. ## experiments in this section, we show and discuss the experimental results. to evaluate the effectiveness of our proposed method, we conducted extensive experiments and analyses on 11 datasets. ## conclusion in this paper, we presented clipfit for fine-tuning visual-language models. unlike existing prompt tuning and adapter tuning methods, clipfit does not introduce any external parameters and finetunes clip by updating only bias terms of projection layers in ffns of the text encoder and the image encoder's layernorm. to understand the effect of clipfit fine-tuning on the pre-trained model, we conducted various analyses focusing on changes in internal parameters and representations. we conducted extensive experiments and analysis to evaluate clipfit on 11 datasets, whose performances show the superiority of our method.