Hard Prompts Made Interpretable: Sparse Entropy Regularization for Prompt Tuning with RL (2024)

Abstract: With the advent of foundation models, prompt tuning has positioned itself as an important technique for directing model behaviors and eliciting desired responses. Prompt tuning regards selecting appropriate keywords included into the input, thereby adapting to the downstream task without adjusting or fine-tuning the model parameters. There is a wide range of work in prompt tuning, from approaches that directly harness the backpropagated gradient signals from the model, to those employing black-box optimization such as reinforcement learning (RL) methods. Our primary focus is on RLPrompt, which aims to find optimal prompt tokens leveraging soft Q-learning. While the results show promise, we have observed that the prompts frequently appear unnatural, which impedes their interpretability. We address this limitation by using sparse Tsallis entropy regularization, a principled approach to filtering out unlikely tokens from consideration. We extensively evaluate our approach across various tasks, including few-shot text classification, unsupervised text style transfer, and textual inversion from images. The results indicate a notable improvement over baselines, highlighting the efficacy of our approach in addressing the challenges of prompt tuning. Moreover, we show that the prompts discovered using our method are more natural and interpretable compared to those from other baselines.

## introduction while the use of large-scale language models (lms) and vision-language models (vlms), pretrained on a massive amount of data, is becoming a dominant paradigm in machine learning @xcite @xcite , fine-tuning the model parameters for adaptation to downstream tasks requires a vast amount of computational time and resources. in this sense, prompt tuning has emerged as a promising low-cost solution @xcite , discovering input prompts that effectively guide the pre-trained models to generate the desired outputs, while keeping the model parameters frozen. prompt tuning is generally categorized into two approaches, soft and hard prompting methods, based on their representation of prompts. soft prompt methods @xcite primarily focus on learning continuous embedding vectors at the token level, which are called as soft prompts. they usually perform the gradient descent to optimize these continuous embedding vectors @xcite . however, the prompts learned through soft tuning are opaque to human interpretation and are not compatible with other pre-trained models that do not share the same embedding spaces. moreover, computing internal gradients for models is highly resource-intensive, especially as the number of model parameters increases, or even infeasible in cases where models are accessible only through apis @xcite . these limitations necessitate an alternative approach: discovery of the prompts composed of human-readable discrete tokens, referred to as hard prompts @xcite @xcite . hard prompts offer numerous advantages over soft prompts: they are transferable from one pretrained model to another since they are agnostic to the embedding. moreover, they confer compositional control by facilitating manual merging and modification @xcite . despite the benefits, they require large-scale discrete optimization in principle. approaches for finding optimal hard prompts often employ backpropagated gradients from models as with soft prompt methods, while circumventing the discrete optimization by, roughly speaking, mapping the soft prompt to the similar discrete tokens in the embedding space @xcite . however, particularly when the model is black-box, reinforcement learning (rl) serves as a powerful alternative tool for optimization, as exemplified by rlprompt @xcite that uses soft q-learning @xcite . one of the key ideas behind rlprompt is an efficient parameterization leveraging a frozen pretrained lm. however, as we show later in the paper, this is accompanied by detrimental approximation error, leading to undesirable results. in this paper, we address this limitation in a principled manner. our approach leverages sparse tsallis entropy regularization for rl @xcite to ignore very unlikely tokens from consideration. we demonstrate the effectiveness of our algorithm across various tasks, including few-shot text classification, unsupervised text style transfer, and textual inversion from images, comparing it against various baselines. most importantly, unlike hard prompts learned by baselines which are often referred to as the 'secret language' of models due to their opacity for human interpretation, our learned prompts are more natural and straightforward. our contributions are outlined as follows: however, @xcite pinpointed a limitation of this approach: the resulting embedding sequences often lack clear semantic interpretation. to address these limitations, our work focuses on the hard prompts optimization within a selected set of tokens, thereby generating task-specific and interpretable tokens. discrete optimization for hard prompts au-toprompt @xcite ) is an initial framework for discrete prompt optimization in transformerbased language models, inspiring a range of diverse methods. these include a gradient-free phrase editing method @xcite , a reinforcement learning-based approach @xcite , and an embedding optimization approach based on langevin dynamics @xcite . in our work, we first benchmark gradient-based methods like autoprompt @xcite and pez @xcite for hard prompt tuning. autoprompt employs the hotflip algorithm @xcite to greedily identifies optimal tokens for each position based on model gradients, while pez performs gradient re-projection during its continuous optimization within the embedding space. however, both methods require substantial computational cost in calculating gradients and are unsuitable for black-box models. on the other hand, rl-prompt @xcite employs a gradient-free rl-based method, serving as our primary baseline. rlprompt introduces an efficiently parameterized network that maps continuous embedding vectors to adaptive vectors within the same space. despite its simplicity, rlprompt struggles with accurately representing q-values across all tokens, potentially leading to sub-optimal prompts as we discuss in section 4.1. ## preliminaries hard prompt tuning with rl hard prompt tuning is the process of discovering an optimal prompt within the token space v, to efficiently tackle specific downstream tasks. assuming a fixed-length prompt of l tokens, this optimization can be formally defined as rl problem: here, the objective is to find a discrete prompt z from the solution space of length-l token sequences v l , which maximizes a task-specific reward function r when concatenated with input x. this reward function measures the appropriateness of the model output y(z, x), i.e., the output from lms or vlms when prompted with z for input x. for example, in a few-shot text classification task utilizing masked lms as task model, the reward function can be defined as a binary signal that indicate the correctness of model output based on available few-shot data, where model output refers to the predicted class for the [mask] token position foot_1 . eq. ( 1 ) can be approached using a bandit algorithm, which aims to identify a length-l token sequence z that maximizes the reward r without gradient information. however, to cope with the exponentially large action space o(|v| l ) for the bandit algorithm, we can treat the optimization as a sequential decision-making process, as in rl-prompt @xcite . more concretely, at each time step t, the algorithm chooses the token z t based on the tokens z 0:t-1 chosen at previous time steps, denoted as policy π(z t |z 0:t-1 ). the calculation of reward r is delayed until the completion of the entire prompt sequence z to obtain the model output. thus, we optimize the policy π with the reformulation of eq. ( 1 ), given by soft q-learning (sql) and rlprompt rl-prompt @xcite employs sql @xcite , an rl algorithm that incorporates entropy regularization. it aims to maximize the expected cumulative reward with the bonus given by the entropy of the action distribution in order to balance the trade-off between exploitation and exploration. more formally, at each time step t, rlprompt trains the policy with the objective: eq. ( 2 ) yields an analytical solution for the optimal policy, expressed as π * (z|z 0:t-1 ) = softmax q(z 0:t-1 , z) α . however, since the action-value function q is not readily available, it is estimated by a neural network parameterized by θ, referred to as q-network. one of the main contributions of rlprompt is the introduction of an efficient parameterization for the q-network. this involves integrating a frozen and pre-trained language model (lm), referred to as the policy lm, into the lower layers of the qnetwork. besides, a trainable multi-layer perceptron (mlp) layer is augmented at the upper level for the adaptation to the downstream task. thus, the trainable parameter θ of the q-network is only the parameters of the mlp layer. formally, given the prefix of prompt z 0:t-1 , the encoding vector e @xmath0, where e denote the real vector space of embeddings. subsequently, this adapted embedding êt is multiplied with the lmhead matrix of policy lm, w @xmath1, to get the next prompt token probabilities. here, the target value q is the bootstrapped estimate of the action value, given by: sparse tsallis entropy regularized q-learning given a policy π(•|z 0:t-1 ) that represents a probability distribution on z, the tsallis entropy @xcite with entropic index q is defined as s q (π) = k 1 q-1 (1z π q (z|z 0:t-1 )), where k is a positive scalar value. it generalizes various types of entropy via the entropic index q. specifically, as @xmath2, it becomes shannon entropy, s 1 (π) = e π [-log π(z|z 0:t-1 )], which is employed by sql for regularizing policy. this results in an optimal policy that is softmax function over q-values, a feature known to promote exploration within the decision-making. however, as mentioned earlier, an intrinsic characteristic of a softmax policy is its distribution of nonzero probability mass across all actions. on the other hand, @xmath3, s 2 (π) = e π [ 1 2 (1π(z|z 0:t-1 ))], which is the focus of this paper. employing sparse tsallis entropy as regularization @xcite leads to an sparse optimal policy that concentrates probability mass on a subset set of actions. analogous to eq. ( 2 ), the analytical formula of the optimal policy with the sparse tsallis entropy regularization with the regularization coefficient α is given by with z (m) indicating the action with the m-th largest value of q(z 0:t-1 , z). thus, the supporting set s q(z 0:t-1 ,•)/α consists of top-k α tokens with the highest action values, and the coefficient α controls the cardinality k α := |s q(z 0:t-1 ,•)/α |. consequently, the sparse policy from eq. ( 4 ) assigns non-zero probabilities for only k α -actions, where smaller α makes the policy sparser. for training q θ (z 0:t-1 , z), we minimize the temporal difference error using the target value q: ## limitations pin can discover prompts that are more interpretable compared to baselines. however, we acknowledge a couple of inherent limitations. firstly, the algorithm exhibits a relatively higher time consumption for training when compared to gradientbased methods. secondly, despite its advanced capabilities in discovering interpretable prompts, our algorithm does not guarantee the consistent discovery of grammatically perfect sentences. we would leave discovering grammatically perfect prompts as future work. ## conclusion in this paper, we firstly discuss the overdetermined issue encounterd in the efficiently parameterized network. to address this issue, we propose pin algorithm, which uses sparse tsallis entropy regularization to systematically exclude ignorable tokens from constraints. prompts learned by pin exhibit better performance in various tasks. future work could explore alternative strategies for identifying ignorable tokens to improve interpretability further, like leveraging task-specific domain knowledge.