S table P rompt : Automatic Prompt Tuning using Reinforcement Learning for Large Language Model (2024)

Abstract: Finding appropriate prompts for the specific task has become an important issue as the usage of Large Language Models (LLM) have expanded. However, the variety of input-output formats complicate finding the prompts. Reinforcement Learning (RL) is a promising for prompt tuning due to its ability to incrementally produce better results through interaction with the environment. But its inherent training instability and environmental dependency make it difficult to use in practice. In this paper, we propose StablePrompt, a prompt tuning method based on RL. We formulate prompt tuning as RL problem between agent and target LLM, and introduce Adaptive Proximal Policy Optimization (APPO), an modified version of PPO for prompt tuning. APPO introduces an anchor model and updates it adaptively based on the training trajectory. Using this anchor model for the KL divergence term in PPO keeps the search space flexible and ensures training stability. We evaluate StablePrompt on various tasks, including text classification, question answering, and text generation. StablePrompt achieves State-of-The-Art performance across diverse tasks. We demonstrates that StablePrompt performs well across various types and sizes of LLMs. Furthermore, we present TTE-StablePrompt, an extension for generating input-dependent prompts. It outperforms StablePrompt in tasks that are hard to solve with a single prompt. This shows that StablePrompt is an extensible and stable RL framework for LLM.

## introduction from semantics @xcite to gpt-4 @xcite , language models have undergone a huge evolution. recently, large language models (llm) are not only used in traditional natural language processing tasks such as text classification @xcite and summarization @xcite , but are also being applied to a wider range of tasks including question answering @xcite , chatting @xcite , math problem solving @xcite , and planning @xcite . while llms perform well in these areas, they rely heavily on handcrafted prompts. finding or tuning the prompts automatically is crucial to use and evaluate the ability of llms in a wider range of applications. reinforcement learning (rl) is a prominent method for prompt tuning due to its ability to update prompts without gradients in the discrete word space. however, rl is vulnerable to overfitting and is highly dependent on the environment. this challenge limits the use of rl for a wide variety of llms and tasks previous methods addressed this by limiting the prompt length @xcite or constraining the action space @xcite , but these approaches reduce the performance of the prompt due to the restricted search space. in this paper, we propose stableprompt that keeps training stability while ensuring search space flexibility. we define prompt tuning as an online, on-policy rl problem for a given dataset and target llm. stableprompt sets the agent model as the llm and optimizes the agent model with adaptive proximal policy optimization (appo). appo adaptively adjusts the policy update rate by introducing an anchor model, a snapshot of a point in time on the training trajectory. this leverages the strong language understanding capabilities of the pre-trained llms to give the agent model search space flexibility while maintaining training stability. we propose two prompt tuning frameworks using appo: stableprompt and test-time editing stableprompt (tte-stableprompt). stableprompt generates a single prompt appropriate for the entire dataset, while tte-stableprompt generates appropriate prompts for each input. we validate our methods on a variety of tasks and llms. the datasets include text classification @xcite ), text understanding @xcite , question answering @xcite , and instruction induction @xcite . for agent and target llms, we conduct experiments on different sizes ranging from 2b to 13b, using various models such as llama @xcite , @xcite , gemma @xcite , and falcon @xcite . to the best of our knowledge, our method is the first rl-based approach that works on agents llm larger than 7b. stableprompt achieves state-of-the-art performance across various tasks. our contributions are summarized as follows: appo introduces an anchor model and modifies the kl-divergence term to keep training stable while ensuring the search space is flexible. ## conclusion in this paper, we propose a novel rl-based prompt tuning method, stableprompt. stableprompt defines prompt tuning as an online, on-policy rl problem and introduces appo. we validate that stableprompt outperforms than other methods across various target models and tasks. to the best of our knowledge, this is the first rl-based prompt tuning method for models larger than 7b. stableprompt demonstrates the potential to integrate existing rl methodologies into prompt tuning, and we believe there is a capacity for further expansion of rl-based prompt tuning approaches.