RLP rompt: Optimizing Discrete Text Prompts with Reinforcement Learning (2022)

Abstract: Prompting has shown impressive success in enabling large pre-trained language models (LMs) to perform diverse NLP tasks, especially with only few downstream data. Automatically finding the optimal prompt for each task, however, is challenging. Most existing work resorts to tuning *soft* prompts (e.g., embeddings) which fall short of interpretability, reusability across LMs, and applicability when gradients are not accessible. *Discrete* prompts, on the other hand, are difficult to optimize, and are often created by “enumeration (e.g., paraphrasing)-then-selection” heuristics that do not explore the prompt space systematically. This paper proposes RLPrompt, an efficient discrete prompt optimization approach with reinforcement learning (RL). RLPrompt formulates a parameter-efficient policy network that generates the optimized discrete prompt after training with reward. To harness the complex and stochastic reward signals from the large LM environment, we incorporate effective reward stabilization that substantially enhances training efficiency. RLPrompt is flexibly applicable to different types of LMs, such as masked (e.g., BERT) and left-to-right models (e.g., GPTs), for both classification and generation tasks. Experiments on few-shot classification and unsupervised text style transfer show superior performance over a wide range of existing fine-tuning or prompting methods. Interestingly, the resulting optimized prompts are often ungrammatical gibberish text; and surprisingly, those gibberish prompts are transferrable between different LMs to retain significant performance, indicating that LM prompting may not follow human language patterns.

## introduction prompting has emerged as a promising approach to solving a wide range of nlp problems using large pre-trained language models (lms), including leftto-right models such as gpts @xcite and masked lms such as bert @xcite , roberta @xcite , etc. compared to conventional fine-tuning that expensively updates the massive lm parameters for each downstream task, prompting concatenates the inputs with an additional piece of text that steers the lm to produce the desired outputs. a key question with prompting is how to find the optimal prompts to improve the lm's performance on various tasks, often with only a few training examples. one of the most popular scheme is to tune soft prompts (i.e., continuous embedding vectors) as they are amenable to gradient descent @xcite @xcite @xcite @xcite . however, the resulting prompts are, by their nature, hard for humans to understand @xcite @xcite and incompatible for use with other lms. besides, the required lm internal gradients are often expensive to compute, or simply unavailable for lms deployed with only inference apis (e.g., gpt-3). it is thus often desirable to use discrete prompts which consist of concrete tokens from a vocabulary. however, their discrete nature renders the optimization very difficult. previous work has typically relied on manual engineering @xcite @xcite , or selecting from multiple paraphrased/generated prompts @xcite @xcite @xcite . autoprompt @xcite uses gradient information to edit the prompt tokens, which suffers from training instability as well as the same applicability issue as gradient-based soft prompting, showing limited effectiveness in practice. methods frozen lms automated gradient-free guided optimize few-shot zero-shot transferrable b/w lms interpret--ability fine-tuning ✗ ✓ ✗ ✓ ✗ ✗ ✗ ✗ manual prompt ✓ ✗ ✓ ✗ ✓ ✓ ✓ ✓ instructions ✓ ✗ ✓ ✗ ✓ ✓ ✓ ✓ in-context demonstration ✓ ✓ ✓ ✗ ✓ ✗ ✓ ✓ soft prompt tuning ✓ ✓ ✗ ✓ ✓ ✗ ✗ ✗ discrete prompt enumeration ✓ ✓ ✓ ✗ ✓ ✓ ✓ ✓ autoprompt (shin et al., 2020) ✓ ✓ ✗ ✓ ✓ ✗ ✓ ✓ rlprompt (ours) ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ on the other hand, rl for prompt optimization poses new challenges to learning efficiency: the large black-box lm presents a highly complex environment that, given the prompt (i.e., actions), goes through a long series of complex transitions (e.g., reading the input and inferring the output) before computing the rewards. this makes the reward signals extremely unstable and hard to learn from. to overcome this difficulty, we propose two simple yet surprisingly effective ways to stabilize the rewards and improve the optimization efficiency. experiments on few-shot classification and unsupervised text style transfer show our approach improves over a wide range of fine-tuning and prompting methods (e.g., those described in in particular, the optimized prompts, though inducing strong task performance, tend to be gibberish text without clear human-understandable meaning, echoing recent research @xcite @xcite that lms making use of prompts do not necessarily follow human language patterns. perhaps surprisingly, those gibberish prompts learned with one lm can be used in other lms for significant performance, indicating that those different pre-trained lms have grasped shared structures for prompting. ## discrete prompt optimization with rl we present rlprompt, a framework for learning prompts of discrete tokens for pre-trained lms to succeed in a wide range of nlp tasks. as discussed in §1, discrete prompts can be easier to interpret and use than continuous prompts, but also more challenging to learn due to intractable optimization over discrete tokens. to solve this difficulty, we formulate discrete prompt optimization as an rl problem, using a continuous policy network to explore the prompt space. the network is highly parameter-efficient, only training a small mlp over a frozen compact lm (e.g., distilgpt-2). below, we present our rl formulation of discrete prompt optimization ( §2.1-2.2). we then discuss the design of our policy network ( §2.3). finally, we describe our reward engineering techniques to improve rl training ( §2.4). ## experiments the proposed rlprompt is generally applicable to various types of lms for performing different nlp tasks using diverse prompt formats (figure 1 ). we evaluate our approach on both classification (in few-shot setting, §3.1) and generation (unsupervised text style transfer, §3.2), and perform rich analyses for new insights on lm prompting ( §3.3). we will release all code and data upon acceptance. ## related work we discuss briefly the various prompting paradigms in previous work, and provide more comprehen-sive discussion in appendix §c. the conventional usage for pre-trained lms is fine-tuning on downstream datasets @xcite , which expensively updates all model parameters and shows limited success with small datasets. @xcite show that manual prompts can steer large lms to perform nlp tasks without any training @xcite @xcite . another line of work @xcite @xcite develop instructional prompts which provide task descriptions instead of fill-in-the-blank questions. with few-shot training examples, @xcite and follow-ups @xcite @xcite achieve remarkable performance by inserting in-context demonstrations. replacing discrete prompts with continuous embeddings, several works @xcite @xcite tune soft prompts using gradient descent. by their continuous nature, however, soft prompts are difficult to understand @xcite @xcite , require expensive gradient information @xcite and are incompatible for reuse across models due to mismatched latent spaces @xcite . some existing works seek to locate better discrete prompts by augmenting human-written prompts with heuristics such as paraphrasing @xcite , editing @xcite , and reframing @xcite , and selecting by some downstream metric. autoprompt @xcite edits discrete prompts with guidance from model gradients, which sees some success with large training data but limited general applicability due to unstable approximations. ## conclusion we have presented rlprompt, an efficient and flexible approach for discrete prompt optimization using rl, which improves over a wide range of fine-tuning and prompting methods in experiments on few-shot classification and unsupervised text style transfer. analysis reveals that strong optimized prompts are incoherent but transferrable between lms for remarkable performance. the observation opens up many promising possibilities for prompting, such as learning prompts cheaply from smaller models and performing inference with larger models. we are excited to explore further.