Clip-Tuning: Towards Derivative-free Prompt Learning with a Mixture of Rewards (2022)

Abstract: Derivative-free prompt learning has emerged as a lightweight alternative to prompt tuning, which only requires model inference to optimize the prompts. However, existing work did not take full advantage of the over-parameterized characteristics of large pre-trained language models (PLMs). In this paper, we propose Clip-Tuning, a simple yet effective method that adopts diverse frozen “thinned” networks of PLMs to obtain *a mixture of rewards* and thus advance the derivative-free prompt learning. The thinned networks consist of all the hidden units that survive a stationary dropout strategy, whose inference predictions reflect an ensemble of partial views over prompted training samples. Our method outperforms previous gradient-free prompt learning methods and achieves parity with gradient-based counterparts on seven language understanding benchmarks under few-shot settings.

## introduction extensive research has shown that prompt tuning achieves parity with model tuning (a.k.a., full finetuning) in few-shot scenarios @xcite . however, prompt tuning that relies on backpropagation from very deep pre-trained transformers requires prohibitive computation and time costs, especially those with billions of parameters @xcite @xcite . meanwhile, for many inference-api-based plms, researchers either do not have full access to model weights due to commercial restrictions, or cannot afford the training of enormous parameters, which significantly limits the power of derivative-based tuning. therefore, derivative-free prompt learning has been a promising alternative @xcite . by treating plms as black boxes, derivative-free approaches seem to be a feasible solution to harnessing large plms. @xcite leveraged evolutionary algorithms to optimize the continuous prompts by repeatedly calling the inference apis of plms. it adopts the model performance over a spot of samples as the optimization feedback. nevertheless, few-shot demonstrations can only result in sparse reward, preventing the prompts from taking sufficient informative signals. hence, our goal is to acquire a mixture of diversified rewards for prompt optimization, using colossal plms with millions or billions of parameters in few-shot settings. recent work on lottery ticket hypothesis @xcite states that an over-parameterized plms contain matching subnetworks capable of reaching the full test performance comparable to the original model. then, @xcite ; @xcite find that ensembling a mixture of subnetworks can improve the diversity of model predictions and achieve the performance gain. as such, the ensemble of subnetwork predictions is a particularly interesting setting to increase the diversity of learning signals for derivative-free prompt optimization. since derivative-free prompt learning only conducts model forward pass without backpropagation, clipping a large proportion of model weights will heavily hurt the overall performance. @xcite states that applying dropout to a network amounts to "sampling" a thinned network from it. moreover, previous work @xcite finds that standard dropout can act as "minimal data augmentation" to construct different sample representations. therefore, we employ dropout during model inference to "sample" different "thinned" subnetworks and diversify the data representations. note that our subnetworks are deterministic, while the original dropout is random at each time. for each training example, diverse subnetworks produce a variety of hidden representations and fitness rewards, which diversifies the learning feedback for gradient-free prompt learning. contributions (1) we propose a simple yet effective method clip-tuning, in which multiple frozen subnetworks act as multi-view critics and provide a mixture of informative rewards for gradient-free prompt optimization ( §4.1). the importance and originality of this study are that it explores the new direction of the exploitation of reward diversity in gradient-free prompt optimization. (2) empirical results show that our method has surpassed previous gradient-free prompt learning approaches on seven natural language understanding (nlu) benchmarks in few-shot settings ( §5). surprisingly, the random search method can serve as an excellent few-shot baseline to prime large plms. (3) our method sheds light on inference-only plms and can be a good fit for commercial plm providers to build api-based features. note that our method requires api providers to support dropout operation, whereas api users do not need to make any change based on derivative-free prompt learning. ## derivative-free prompt learning vanilla derivative-free prompt learning @xcite employs the model inference to evaluate the fitness of candidate prompts for iterative prompt learning using evolutionary algorithms. firstly, it prepends a series of soft prompt embeddings p to the input tokens x to feed the prompted input [p ; x] into the frozen pre-trained transformers f parameterized by θ. the prompt @xmath0. meanwhile, @xcite find that plms have a low dimension reparameterization. the search space of p ∆ can be reparameterized as a intrinsic dimensionality @xmath1; that is, p ∆ = z • w. thus, the prompt is optimized over an intrinsic dimensionality of z instead of p ∆ . then it performs task-specific inference to get the fitness of candidate prompts using an arbitrary objective function l(f θ ([p ; x]), y), where y denotes the golden standard label for the input x. afterward, the cma-es algorithm iteratively optimizes the prompt fitness to get the optimum prompt that satisfies arg min p l(f θ ([p ; x]), y). ## conclusion and future work we propose clip-tuning, a simple method to enhance derivative-free prompt learning for nlu tasks. it focuses on acquiring fine-grained informative feedback for derivative-free optimization, beating the previous rivals on seven nlu benchmarks. our method sheds light on the exploration of inference-only api-based plms. in the future, it is a promising direction to reduce the cost of green derivative-free prompt optimization.