Reliable Gradient-free and Likelihood-free Prompt Tuning (2023)

Abstract: Due to privacy or commercial constraints, large pre-trained language models (PLMs) are often offered as black-box APIs. Fine-tuning such models to downstream tasks is challenging because one can neither access the model’s internal representations nor propagate gradients through it. This paper addresses these challenges by developing techniques for adapting PLMs with only API access. Building on recent work on soft prompt tuning, we develop methods to tune the soft prompts without requiring gradient computation. Further, we develop extensions that in addition to not requiring gradients also do not need to access any internal representation of the PLM beyond the input embeddings. Moreover, instead of learning a single prompt, our methods learn a distribution over prompts allowing us to quantify predictive uncertainty. Ours is the first work to consider uncertainty in prompts when only having API access to the PLM. Finally, through extensive experiments, we carefully vet the proposed methods and find them competitive with (and sometimes even improving on) gradient-based approaches with full access to the PLM.

## introduction pre-trained language models (plms) are versatile learners and demonstrate impressive few-shot capabilities @xcite and promising performance @xcite @xcite on various downstream tasks such as text classification @xcite , commonsense reasoning @xcite , question answering @xcite , and machine translation @xcite . the conventional approach to adapting plms to downstream tasks involves fine-tuning the model @xcite . although fine-tuning is effective, it can be challenging to do in practice. first, fine-tuning large language models are compute and memory intensive, e.g., a large model like gpt-3 @xcite ) contains billions of parameters. further, it is inefficient to adapt a plm to a large number of downstream tasks since each task would require storing a copy of model parameters. prompt tuning alleviates these issues by providing an efficient way to adapt a plm to a downstream task. it only learns a small number of prompt parameters while keeping the large plm frozen but still achieves comparable performance to fine-tuning the entire plm @xcite @xcite . although more efficient than traditional finetuning, prompt tuning still requires the propagation of gradients through the entire plm. beyond being computationally expensive, this may not be possible due to privacy risks or legal and commercial constraints. in fact, large plms are often only made available in the form of black-box apis @xcite . motivated by these observations, a recent line of research @xcite has started exploring gradient-free approaches to prompt tuning. bbt @xcite optimizes continuous prompt by leveraging the derivative-free optimization algorithms, and bbtv2 @xcite improves over bbt by optimizing multiple deep prompts at various intermediate layers of plm. although these approaches are gradient-free, they still assume that intermediate layers of the model being tuned are accessible. moreover, when deploying an nlp model in a real-world setting, it is inevitable to encounter unexpected scenarios. for example, the test data to be predicted might originate from out-of-distribution resources @xcite . for the model to be useful in such scenarios, it is essential that the model is able to quantify the uncertainty associated with its predictions and that these uncertainties are well-calibrated. to this end, here we further push the limits of gradient-free prompt tuning in two aspects: gradient-free prompt tuning gradient-free prompt tuning aims to learn the continuous prompt without the propagating gradients through the plm. bbt @xcite utilizes derivative-free optimization algorithms to optimize the continuous prompt. bbtv2 @xcite extends bbt by incorporating the idea of deep prompt tuning, which optimizes the deep prompt injected at additional intermediate layers of the plm. since our goal is to treat the plm as a black-box, deep prompt tuning is out of the scope of this work. we instead focus on the problem setting of the original bbt @xcite ) that learns a single prompt at the input layer. beyond point-estimates of prompts many applications demand accurate quantification of uncertainty in predictions. this can be achieved in the prompt-tuning setting by not just learning a point estimate of the prompts but also inferring a distribution over the prompts for a given downstream task. in a non-black-box setting, to infer such a distribution, we can apply classical frequentist or bayesian approaches. although a few recent works focus on uncertainty quantification in nlp applications @xcite @xcite , quantifying uncertainty in prompt-tuned large language models remains a severely under explored area. our paper is the first to explore prompt uncertainty in gradient-free settings. simulation-based inference classic approaches for statistical inference mentioned above are intractable when the likelihood function is not accessible. the problem of inferring parameters of such a black-box model, called simulation-based inference (sbi) @xcite , is gaining popularity. traditional sbi approaches include approximate bayesian computation (abc) @xcite @xcite @xcite and synthetic likelihood (sl) @xcite . more recently, the neural density estimation-based approaches utilize the powerful deep neural network density estimator to directly learn the likelihood, i.e., sequential neural likelihood estimation (snle) @xcite , or the likelihood ratio, i.e., sequential neural ratio estimation (snre) @xcite , or the posterior, i.e., sequential neural posterior estimation (snpe) @xcite . ## problem formulation in this paper, we focus on text classification and restrict ourselves to the few-shot learning setting considered in bbt @xcite . given a dataset @xmath0, y) = {(x i , y i )} n @xmath1, we aim to adapt f to predict the label y * for an unseen text pas-figure 1 : our general goal is to estimate the posterior distribution of prompts. the gradient-free setting uses the internal logits of plm for optimization. our proposed gradient-free variational inference approach utilizes the likelihood to compute the elbo objective and leverage the gradient-free optimizer to optimize the variational distribution. the gradient-free and likelihood-free setting can be formulated as an sbi problem, where the plm is treated as a black-box simulator, and its output discrete outcome labels are the simulated data. the posterior samples can be efficiently approximated by the proposed abc-smc algorithm. sage x * . we formulate the classification task as a masked language modeling problem, where the input text x i is converted into xi via predefined templates, e.g., adding trigger words like "it was [mask]", and the labels y i are mapped to label tokens ỹi in the vocabulary such as "great" or "bad". we denote this transformed dataset @xmath2, ỹ). we use soft prompt tuning @xcite to adapt f , i.e., we construct a continuous prompt embedding @xmath3, ŷi = f ( xi ; p), where the notation ŷi = f ( xi ; p) is short hand for ŷi ∼ cat(σ(h θ ( xi ; p))). here, cat denotes the categorical distribution, σ is the softmax function, and θ represents the frozen parameters of the plm. we use h θ to denote all but the final layer of the plm f . finally, we aim to learn an optimal prompt when we are unable to propagate gradients through f , we need to rely on gradient-free approaches to optimize equation 1. recent work @xcite has demonstrated promising gradientfree prompt tuning results by first employing a lower dimensional re-parameterization, @xmath4, @xmath5, where @xmath6, and then using gradientfree evolutionary algorithms, in particular, covariance matrix adaptation evolution strategy (cma-es) @xcite to optimize, going forward, we also adopt this lower dimensional parameterization, but instead of learning a point estimate z * , we learn a distribution p(z| d) in a gradient-free setting. similar to the point estimated variants, our algorithms to learn p(z| d) also rely on cma-es. next, we consider the fully black-box setting -likelihood-free and gradient-free. here, beyond being unable to propagate gradients through f , we are further handicapped by only observing the predicted label tokens, ŷi = f ( xi ; p) for each training instance xi , and not the corresponding logits, i.e., h θ ( xi ; p). in this more challenging setting we found cma-es based approaches to be unreliable, often getting stuck in poor optima. instead, we found it effective to pose the likelihood-free and gradient-free prompt tuning task as a simulationbased inference (sbi) @xcite problem. we view the plm f as a black-box simulator that given a realization of z and the text xi produces ŷi . we then use a sequential monte-carlo approximate bayesian computation (smc-abc) approach to infer the distribution p(z| d). finally, we use the distribution p(z| d) to characterize the uncertainty in predictions via the predictive distribution p(ỹ| x, d) = p(ỹ| x; z)p(z| d)dz. we form monte-carlo approximations to this integral. in the gradient-free case, this is, ## methods we now describe our methods in greater detail. first, we discuss two algorithms for the gradientfree setting in 4.1 and 4.2. after that, we focus on addressing the gradient-free and likelihood-free setting from the sbi perspective in 4.3. ## experiment results in this section, we demonstrate the solid empirical performance of our proposed methods. we begin with introducing the uncertainty quantification applications and describe the experiment settings. then, we present our main results in terms of prediction performance and uq quality. finally, we provide an ablation study and relevant perspectives of comparison. detailed results and implementation steps are provided in appendix d. benchmark. for a comprehensive comparison with bbt @xcite , we mainly employ the same text classification benchmark datasets as bbt, including sentiment analysis datasets sst-2 @xcite and yelp polarity @xcite ; topic classification datasets ag's news @xcite and dbpedia @xcite ; paraphrase dataset mrpc @xcite ; natural language inference (nli) datasets snli @xcite and rte @xcite . both calibration and selective classification tasks are conducted using the original test samples for each benchmark dataset. for the far ood detection task, we create the id/ood dataset pairs by combining two datasets belonging to two different tasks, e.g., sst-2/rte. for the near ood detection task, we use imdb @xcite for the sentiment analysis task and mnli @xcite for the nli task. baselines. for prediction performance, besides the sota gradient-free prompt tuning approach bbt @xcite , we also compare with other gradient-free methods: (1) the naive manual prompt that uses the hand-crafted prompt templates; (2) in-context learning @xcite ; (3) feature-based approaches @xcite that trains auxiliary models on top of the plm extracted features, including feature-mlp training a mlp classifier and feature-bilstm @xcite that trains the several continuous prompts injected at different layers of plm. for uncertainty quantification tasks, few existing prompt tuning works aim to tackle this problem, so we mainly compare with bbt to justify how we can address its limitation under the gradient-free setting. implementation details. we follow the same experiment setting as bbt. we focus on text classification as a few-shot learning problem, motivated by the fact that labeled training data can be limited in practice. specifically, we construct few-shot training and validation data by drawing 16 random samples for each class from the original training dataset. the prediction performance is evaluated on the original development or test set, depending on the datasets. we use the same plm model roberta large as the backbone model and keep the hyper-parameter same as bbt. specifically, we set the prompt length as 50, i.e., @xmath7, and the subspace dimensionality as @xmath8. the only modification is that we adapt the normal distribution @xcite to generate the random projection matrix a, instead of the uniform distribution used in bbt. for a fair comparison, we reproduce the results of bbt using the random pro-jection generated from normal distribution. more implementation details are included in appendix c. performance metrics. for prediction performance, we evaluate the prediction accuracy on the testing dataset. for calibration performance, we adopt the expected calibration error (ece) @xcite score as the metric. for both selective classification and ood detection tasks, we compute the area under the risk vs. rejection rate curve (aurrrc) @xcite . the risk is defined as the portion of wrong-predicted samples among the data chosen for prediction in selective classification and the portion of ood samples among the data identified as in-distribution in the ood detection task. the rejection rate is defined as the portion of data that abstained from the prediction based on specific uncertainty measurement. note that an oracle with perfect knowledge of uncertainty measurement can achieve a minimum aurrrc score. this is obtained by assigning an uncertainty score based on the oracle knowledge, i.e., whether a test sample is wrongpredicted (ood samples) or not. we denote such minimum aurrrc score as the lower-bound. given the predictive label distribution, we utilize two uncertainty measurements, including entropy of the label distribution, i.e., h p(ỹ| x; d) , and maxp, which is defined as max c p(ỹ = c| x; d). ## concluding remarks in this work, we explore gradient-free prompt tuning along two under-explored angles: quantifying uncertainty in soft prompts; and tackling a more strict likelihood-free setting from the sbi perspective. our developed methods demonstrate encouraging empirical performance across multiple tasks. investigating more modern neural sbi methods and designing more robust methods for learning prompt posteriors are exciting directions for future research. other perspectives on gradient-free prompt tuning, such as learning natural languagelike interpretable prompts, are also worthy of exploration.