When Gradient Descent Meets Derivative-Free Optimization: A Match Made in Black-Box Scenario (2023)

Abstract: Large pre-trained language models (PLMs) have garnered significant attention for their versatility and potential for solving a wide spectrum of natural language processing (NLP) tasks. However, the cost of running these PLMs may be prohibitive. Furthermore, PLMs may not be open-sourced due to commercial considerations and potential risks of misuse, such as GPT-3. The parameters and gradients of PLMs are unavailable in this scenario. To solve the issue, black-box tuning has been proposed, which utilizes derivative-free optimization (DFO), instead of gradient descent, for training task-specific continuous prompts. However, these gradient-free methods still exhibit a significant gap compared to gradient-based methods. In this paper, we introduce gradient descent into black-box tuning scenario through knowledge distillation. Furthermore, we propose a novel method GDFO, which integrates gradient descent and derivative-free optimization to optimize task-specific continuous prompts in a harmonized manner. Experimental results show that GDFO can achieve significant performance gains over previous state-of-the-art methods.

## introduction large pre-trained language models (plms) @xcite @xcite have attracted considerable attention for their versatility and potential for solving a wide spectrum of natural language processing (nlp) tasks. especially, through prompt-based learning (pl) @xcite , plms have consistently demonstrated impressive performance on various downstream tasks with a figure 1 : accuracy (%) on the ag's news and dbpedia datasets. experimental setup is detailed in section 4.3. it is noted that prior gradient-based approaches, such as adapter @xcite and @xcite , are unable to be used in black-box scenarios. gdfo is the first to introduce gradient descent in the black-box tuning scenario. few labeled samples. however, it is a challenge to extend the benefits of these large plms to a broader audience. for users, the cost of running these models may be prohibitive; for service providers, they may not open source the model parameters due to commercial considerations and potential risks of misuse foot_0 . one possible solution is to deploy plms as a service, enabling users to access the advanced capabilities of plms through their inference apis, such as gpt-3 @xcite , ernie @xcite and yuan @xcite . in this scenario, the large pre-trained language model provided by the server is considered as a black box. in order to perform various downstream tasks, users are required to construct task-specific prompts or select training samples @xcite to input into the black box. we can manually construct discrete prompts, which are simple and effective but may not fully utilize training data, potentially resulting in suboptimal performance on some tasks. instead of designing hand-crafted discrete prompts, there have been an increasing number of studies on continuous prompt tuning @xcite @xcite , which aim to train continuous prompts and add them to the original samples. trainable continuous prompts have also shown remarkable success on various tasks, but most existing methods optimize the continuous prompts through back-propagation, which is unavailable in the black-box scenario. to solve the issue, @xcite have recently proposed black box tuning (bbt), which utilizes random projection matrices and derivative-free optimization (dfo) @xcite @xcite , instead of gradient descent, for training continuous prompts in the black-box scenario. built upon bbt, bbtv2 @xcite prepends continuous prompts to each layer of the plm and further presents a divide-andconquer gradient-free algorithm to alternately optimize the prompts at different layers. both bbt and bbtv2 have shown their superiority against other gradient-free methods. despite the success, there remains a significant gap compared to gradientbased methods on certain tasks. for example, compared against bbtv2, adapter @xcite , a gradient-based method, leads by 4.35% on the dbpedia dataset (as shown in figure 1 ). therefore, we consider that the incorporation of gradient descent into the black-box scenario may potentially enhance the performance of the model. based on the insights discussed above, in this paper, we introduce gradient descent into the blackbox scenario through knowledge distillation techniques. in particular, we propose a novel approach named gdfo to combine gradient descent with derivative-free optimization, allowing them to jointly optimize task-specific continuous prompts. first, we adopt the technique of knowledge distillation, where a student model is trained to emulate the knowledge of the black-box model, referred to as the teacher model. then, a prompt generator is trained by gradient descent through the student model, while utilizing derivative-free optimization algorithms to optimize continuous task-specific prompts. the continuous prompts generated by the prompt generator and the prompts optimized by the derivative-free algorithm are further integrated to serve as the final prompts. finally, we perform extensive experiments on seven benchmark datasets to show that gdfo can achieve significant performance gains over other state-of-the-art methods. the main contributions of the paper are summarized as follows: ## method in this section, we describe our approach gdfo. we first give an overview of gdfo, which is illustrated in figure 2 . gdfo first trains a student model by aligning its outputs to that of the teacher model (i.e., the black-box model). then, gdfo trains the prompt generator by gradient descent while simultaneously optimizing the continuous prompts via dfo. finally, the final prompts are obtained by integrating the prompts generated by the prompt generator with those optimized by dfo, which are then fed into the black-box model together with query instances to obtain predictions. next, we describe each component of gdfo in detail. ## experiments in this section, we perform comprehensive experiments to compare our proposed model with twelve competitive baselines on seven downstream tasks. ## conclusion in this paper, we introduced gradient descent into the black-box tuning scenario through knowledge distillation for the first time, which provided a novel insight for future black-box tuning approaches. furthermore, we proposed a novel method, gdfo, which integrates gradient descent and derivative-free optimization for jointly training continuous prompts. gdfo first trains a student model to enhance the performance by aligning its outputs with those of the teacher model (i.e., the black-box model). after that, gdfo trains a prompt generator using gradient descent while simultaneously optimizing a continuous prompt using dfo algorithm. experimental results on various datasets show that gdfo can achieve significant performance gains over other gradient-free and gradient-based methods.