Black-Box Tuning of Vision-Language Models with Effective Gradient Approximation (2023)

Abstract: Parameter-efficient fine-tuning (PEFT) methods have provided an effective way for adapting large vision-language models to specific tasks or scenarios. Typically, they learn a very small scale of parameters for pre-trained models in a white-box formulation, which assumes model architectures to be known and parameters to be accessible. However, large models are often not open-source due to considerations of preventing abuse or commercial factors, hence posing a barrier to the deployment of white-box PEFT methods. To alleviate the dependence on model accessibility, we introduce collaborative black-box tuning (CBBT) for both textual prompt optimization and output feature adaptation for black-box models. Specifically, considering that the backpropagation gradients are blocked, we approximate the gradients of textual prompts by analyzing the predictions with perturbed prompts. Secondly, a lightweight adapter is deployed over the output feature of the inaccessible model, further facilitating the model adaptation process. Empowered with these designs, our CBBT is extensively evaluated on eleven downstream benchmarks and achieves remarkable improvements compared to existing black-box VL adaptation methods. Our code will be made publicly available.

## introduction large-scale vision-language (vl) models @xcite @xcite @xcite have demonstrated remarkable performance in a wide range of applications. various model finetuning methods have been proposed to exploit the potential of pre-trained vl models for downstream vision @xcite @xcite @xcite @xcite and natural language processing @xcite tasks. most existing methods conduct parameter-efficient fine-tuning (peft @xcite ), which updates a tiny fraction of the model parameters or introduces a small number of extra parameters for tuning, in order to transfer pre-trained knowledge in a computation-and data-efficient manner. although impressive improvements have been achieved, standard peft methods need to pass signals forward and backward through the entire pre-trained model to update the parameters, which relies on the availability of the architecture, parameters, and even the inference source code of the model. nevertheless, the trend of building machine learning models as a service leads to many proprietary services that only provide an api interface for model inference, e.g., chatgpt, bard, and gpt-4, where the parameters and inference code of the models are not open-source due to commercial or safety considerations. under such black-box circumstances, existing peft methods can hardly be adopted. thus, it is worthwhile to develop methods that can tune pre-trained vl models in a black-box setting. moreover, in the era of large foundation models, running super large pre-trained models on local devices can be very costly as the scale of the pre-trained model has constantly increased. although existing peft methods restrict learnable parameters to a fairly small scale, it is still a burden to accommodate models with billions of parameters in limited computing resources for most users. to tackle these problem of tuning black-box vl models, there exist a few very recent efforts. for instance, blackvip @xcite pioneered black-box prompting for vl models by learning an asymmetric autoencoder-style coordinator with a zeroth-order optimization to modify visual prompts in the pixel space. however, modifying prompts in the large pixel space causes inefficiency and the method requires up to 9k parameters in the coordinator to achieve the goal. besides, the performance of their visual prompts is subject to the diverse semantic features of a well-trained generative selfsupervised learning model. even so, the method demonstrates limited performance improvements after prompting, showing that prompt tuning in the black-box setting is very challenging. in this paper, we propose a collaborative blackbox tuning method dubbed cbbt for tuning pretrained vl models and adapting them to downstream tasks. unlike in blackvip @xcite , we learn the prompt for the textual input instead of images, and we adapt the visual features using an adapter. the basic idea is illustrated in fig. 1 . a query-efficient approximation method @xcite is used to estimate the gradients and optimize the textual prompt with the blackbox pre-trained vl model, from which true gradients are not accessible. specifically, we query the model with randomly perturbed prompts and then summarize the change in model prediction loss to estimate the gradient of learnable parameters (i.e., the prompts). we equip single-step gradient optimization with information from history updates via a momentum strategy, which leads to faster convergence and better results. under the circumstance where the output features are available for the pre-trained vl models, we further adapt the visual features by introducing a lightweight adapter module. as demonstrated in fig. 1 , the visual adapter can be learned effortlessly by supervised learning, without having knowledge of the pre-trained vl backbone. with the joint optimization of the textual prompt and the visual adapter, our cbbt achieves significant model adaptation performance. to evaluate its effectiveness, we conduct extensive experiments on eleven downstream benchmarks, showing superior performance compared to existing black-box vl adaptation methods. the main contributions of this work can be summarized as follows: ## related work black-box prompt tuning for large language models. bbt @xcite adopts derivativefree optimization using covariance matrix adaptation evolution strategy (cma-es) @xcite to optimize the prompt in a low-dimensional intrinsic subspace. with this method, the adaptation of large language models works well on natural language tasks, surpassing even the white-box prompting performance. bbtv2 @xcite further enhances the capacity of bbt by using deep prompt tuning. bdpl @xcite tunes a set of discrete prompts for language models by modeling the choice of words in the prompt as a policy of reinforcement learning, and a variance-reduced policy gradient estimator @xcite @xcite is used to optimize the discrete prompt based on loss value. ## conclusion in this paper, we present cbbt, a black-box adaptation approach for vl models. we effectively tune a soft prompt for the text encoder by gradient approximation and jointly learn a lightweight adapter module to transfer the visual features of the pre-trained backbone. equipped with the textual prompt and the visual adapter, our method achieves a collaborative adaptation for both modalities. experiments on various datasets show that our cbbt performs favorably against the state-of-the-art methods.