BBT v2: Towards a Gradient-Free Future with Large Language Models (2022)

Abstract: Most downstream adaptation methods tune all or part of the parameters of pre-trained models (PTMs) through gradient descent, where the tuning cost increases linearly with the growth of the model size.By contrast, gradient-free methods only require the forward computation of the PTM to tune the prompt, retaining the benefits of efficient tuning and deployment.Though, past work on gradient-free tuning often introduces gradient descent to seek a good initialization of prompt and lacks versatility across tasks and PTMs.In this paper, we present BBTv2, an improved version of Black-Box Tuning, to drive PTMs for few-shot learning.We prepend continuous prompts to every layer of the PTM and propose a divide-and-conquer gradient-free algorithm to optimize the prompts at different layers alternately.Extensive experiments across various tasks and PTMs show that BBTv2 can achieve comparable performance to full model tuning and state-of-the-art parameter-efficient methods (e.g., Adapter, LoRA, BitFit, etc.) under few-shot settings while maintaining much fewer tunable parameters.

## introduction the past few years have witnessed remarkable progress of large language models (llms) @xcite @xcite . it has been repeatedly demonstrated that scaling up the model size is promising to achieve better performance. however, the growing model size also leads to a linear increase in tuning cost. finetuning and deploying a separate copy of the llm for each downstream task become prohibitively expensive. to that end, much effort has been devoted to parameter-efficient tuning (pet) @xcite , which only tunes a small portion of parameters while keeping most of the parameters of the llm unchanged. by pet, llms can be specialized to a downstream task at inference time by activating a small number of task-specific parameters. though it is deployment-efficient, tuning the small portion of parameters still requires back-propagation through the entire llm, which is expensive or even infeasible for many practitioners. to make llms benefit a wider range of audiences, a common practice is to release llms as a service and allow users to access the powerful llms through their inference apis @xcite . in such a scenario, called languaged-model-as-a-service (lmaas) @xcite , users cannot access or tune model parameters but can only tune their prompts to accomplish language tasks of interest. @xcite propose to recast downstream tasks as a language modeling task and perform different tasks by conditioning on taskspecific text prompts. further, they demonstrate that llms exhibit an emergent ability of in-context learning, i.e., llms can learn to perform tasks with a few demonstrations provided in the input context without updating parameters. nevertheless, its performance has been shown to highly depend on the choice of the prompt or the demonstrations @xcite and still lags far behind full model tuning. recently, @xcite proposed black-box tuning (bbt), which optimizes continuous prompt by only accessing model inference apis. in contrast to gradient-based tuning that requires expensive back-propagation 1 , bbt only requires model forward computation, which can be highly optimized by acceleration frameworks such as onnx runetime and tensorrt. in addition, the optimization cost of bbt is decoupled from the scale of the model. instead, larger models can be more favorable to bbt due to the lower intrinsic dimensionality @xcite . despite its efficiency superiority and comparable performance to gradient descent, our pilot experiments ( §3) show that bbt lacks versatility across tasks and ptms. in this paper, we present bbtv2, an improved version of bbt, to address these issues. instead of injecting continuous prompt tokens merely in the input layer, bbtv2 prepends prompts to hidden states at every layer of the ptm (termed as deep prompts), incorporating an order of magnitude more parameters to handle more difficult tasks. however, the increased number of parameters also poses a challenge for high-dimensional derivativefree optimization (dfo, @xcite ; @xcite ). fortunately, we show that the forward computation of modern ptms can be decomposed into an additive form w.r.t. hidden states of each layer thanks to the residual connections @xcite . hence, the optimization of the deep prompts can be decomposed into multiple lowdimensional sub-problems, each corresponding to the optimization of prompt at one layer. based on this insight, we propose a divide-and-conquer algorithm to alternately optimize prompt at each layer. for the optimization at each layer, we maintain a random linear transformation that projects the prompt parameters into a low-dimensional subspace and perform dfo in the generated subspace. to generalize bbtv2 to a variety of ptms, we generate the random projections using normal distributions with ptm-related standard deviations. experimental results show that bbtv2 signif1 the computation and storage cost of back-propagation is proportional to the forward compute. more widely used variants of gradient descent, such as adam (kingma and ba, 2015), even require higher compute resources. ## black-box tuning black-box tuning (bbt) @xcite is a derivative-free framework to drive ptms for fewshot learning. in particular, for a batch of training data (x, y ), we first convert the texts x with some pre-defined templates (e.g., "it was [mask]") into x, and the labels y with a pre-defined map into label words ỹ (e.g., "great" and "terrible"). by this, we can formulate various downstream tasks into a general-purpose (masked) language modeling task and utilize the pre-trained (masked) language modeling head to solve them. assume the ptm inference api f takes a continuous prompt p and a batch of converted texts x as input, and outputs the logits of the tokens of interest (e.g., the [mask] token). bbt seeks to find the optimal prompt p ⋆ = arg min @xmath0, x), ỹ ), where p is the prompt space and l is some loss function such as cross entropy. the closed form and the gradients of f are not accessible to bbt. the prompt @xmath1, making it infeasible to be optimized with derivative-free optimization (dfo) algorithms. hence, bbt adopts a random projection @xmath2, i.e., bbt adopts the covariance matrix adaptation evolution strategy (cma-es) @xcite to optimize eq.( 1 ) and obtain the desired prompt p ⋆ = az ⋆ . the random projection a is frozen during optimization. 3 pilot experiments 3.1 limitations of bbt across tasks unsatisfactory performance on entailment tasks. as demonstrated by @xcite , bbt can outperform model tuning on entailment tasks when using pre-trained prompt embedding for initialization. however, pre-trained prompt embedding is not always available for many languages and ptms. without pre-trained prompt embedding, bbt still lags far behind model tuning on entailment tasks. in other words, bbt does not completely get rid of the dependence on gradients to exceed model tuning. in contrast, as depicted in figure 2 , bbtv2 can match the performance of model tuning on three entailment tasks, namely mrpc @xcite , snli @xcite , and rte @xcite without using pre-trained prompt embedding. slow convergence on many-label classification tasks. bbt suffers from slow convergence rate when the number of labels becomes large. as reported by @xcite , bbt cannot converge within the budget of 8,000 api calls, which is sufficient for common tasks to converge, on dbpedia @xcite , a topic classification task with 14 labels. figure 3 shows the cross entropy loss and the training accuracy during optimization. compared with bbt, the proposed bbtv2 significantly accelerates convergence on dbpedia. ## conclusion in this work, we present bbtv2, an improved version of bbt @xcite with deep prompts that are attached to every layer of the ptm. to optimize the high-dimensional prompt parameters, we propose a divide-and-conquer (dc) algorithm combined with random projections to alternately optimize the continuous prompt at each layer. experimental results demonstrate that bbtv2, without any gradient-based component, can achieve comparable performance to state-of-the-art pet methods and full model tuning while maintaining much fewer tunable parameters.