MORL -Prompt: An Empirical Analysis of Multi-Objective Reinforcement Learning for Discrete Prompt Optimization (2024)

Abstract: RL-based techniques can be employed to search for prompts that, when fed into a target language model, maximize a set of user-specified reward functions. However, in many target applications, the natural reward functions are in tension with one another – for example, content preservation vs. style matching in style transfer tasks. Current techniques focus on maximizing the average of reward functions, which does not necessarily lead to prompts that achieve balance across rewards – an issue that has been well-studied in the multi-objective and robust optimization literature. In this paper, we conduct an empirical comparison of several existing multi-objective optimization techniques adapted to this new setting: RL-based discrete prompt optimization. We compare two methods optimizing the volume of the Pareto reward surface and one method that chooses an update direction that benefits all rewards simultaneously. We evaluate performance on two NLP tasks: style transfer and machine translation, each using three competing reward functions. Our experiments demonstrate that multi-objective methods that directly optimize the volume of the Pareto reward surface perform better and achieve a better balance of all rewards than those that attempt to find monotonic update directions.

## introduction discrete prompt tuning involves refining a text prompt for a language model (lm) to maximize a set of user-specified objectives on the lm's output @xcite @xcite . successful techniques for prompt tuning allow users to control and adapt powerful llms to new tasks without the trial-and-error of manual prompt design. while rl-based techniques have been shown to be effective at finding prompts that optimize an average of rewards @xcite , in many target applications, there is a tension between the natural reward functions. for example, as depicted in figure 1 , many style transfer tasks need to preserve content while simultaneously maximizing transfer into the target style -two objectives that are directly at odds with one another. thus, current techniques result in a phenomenon we will refer to as objective collapse: focusing on maximizing the average of reward functions (also called scalarization) can lead to prompts that disproportionately maximize a subset of objectives at the expense of others. for instance, in figure 1 , the prompt on the left side tends to produce lm outputs (represented by blue dots) that prioritize one objective over the other. conversely, the prompt on the right side produces samples that achieve reasonable performance across all objectives simultaneously. however, in both cases the average reward is nearly equivalent. the problem of reward balancing has been well-studied in other domains-for example, the multi-objective and robust optimization literature proposes several approaches that offer advantages figure 2 : in all the settings, we have a parameter-efficient policy model, responsible for generating the task-specific prompts, where all the parameters of the model except for an mlp module are frozen. another frozen language model is used to generate output sentences, given an input and a prompt from the policy model. all the output sentences are then evaluated with respect to each objective, and multi-objective losses are calculated. finally, a gradient update on the mlp parameters is performed. over scalarization. however, these techniques have not been applied to the rl-based discrete prompt optimization setting that is most relevant in nlp. thus, in this paper we conduct an empirical comparison of several existing techniques for multi-objective optimization that we adapt to discrete prompt optimization, where we aim to evaluate their effectiveness in achieving a more useful balance of rewards in downstream promptdriven nlp tasks. the first two approaches we compare maximize the volume of the pareto reward surface, while the third method chooses a gradient update direction that is beneficial for all rewards simultaneously. more specifically, the first method in our study computes the hypervolume indicator (hvi) @xcite for a set of samples drawn from a given prompt, and treats this measure as the final reward in rl. intuitively, hvi measures the area under the pareto frontier of the outputs sampled from the current prompt (shown by the shaded regions in figure 1 ). samples that achieve a better balance of reward elevate the pareto frontier and increase hvi. however, this method has a potential drawback: if an outlier sample (e.g., represented by the red dot labeled with a four in figure 1 ) achieves high values across all rewards, the hvi can be disproportionately high (represented by the outer rectangular region in figure 1 , which dominates the shaded areas). this dominant outlier effect may diminish the stability of hvi optimization in an rl setting, as it becomes very sensitive to outliers. therefore, we also investigate using a simpler method for maximizing the volume in the second approach, called the expected product of rewards. here, we approximate the expected volume by simply computing the average product of rewards (tentatively depicted by the dark rectangular region in figure 1 ). the third approach takes a different strategy based on steepest gradient descent @xcite . we approximate the gradient of the expectation of each individual reward separately and then search for an update direction to make monotonic progress in every reward simultaneously. to understand the effectiveness of these approaches in the discrete prompt optimization setting, we conduct experiments on two text generation tasks: text style transfer and machine translation using three competing reward functions for each task. our findings indicate that volumebased methods are most effective in this setting, achieving substantial gains in balancing the competing rewards, compared to the baseline methods. while rl-based steepest descent also improves balance, it does not perform as robustly as the volume-based methods. ## problem statement in this paper, we specifically focus on optimizing discrete prompts, as they offer the advantages of interpretability and reusability in contrast to continuous or "soft" prompts. we acknowledge that the issue of balancing multiple conflicting objectives is a well-established area of research within the multi-objective and robust optimization literature. however, adapting these techniques to the domain of discrete prompt optimization for language models comes with challenges due to having sources of discontinuity and discreteness. first, the text tokens for the prompt are discrete, and second, since marginalizing over algorithm 1: volume-based policy update for one input sentence. 1: input: input sentence x, policy π θ , reward models r 1...m , external frozen lm 2: {z 1...k } ∼ π θ (x) ## methodology in this section, we describe the adapted optimization methods for generating discrete prompts that, when fed into an lm along with the input text, produce outputs that maximize a set of competing reward functions. we compare two optimization methods that maximize the volume coverage of rewards and one method that finds the gradient update direction which optimizes all rewards simultaneously. we adopt the soft q-learning (sql) reinforcement learning framework introduced by @xcite , which has demonstrated effectiveness in discrete prompt optimization with single reward functions or scalarization @xcite . in line with @xcite , we utilize only the on-policy component of sql. for clarity and simplicity, we omit certain details of the sql updates in the pseudo-codes (figures 3 and 4 ), focusing instead on the novel multi-objective components. for a comprehensive discussion of sql, we refer the reader to @xcite . ## experiments we now describe the empirical comparison of the rl-adapted multi-objective optimization methods that we introduced in the previous section. our primary aim is to evaluate these techniques for discrete prompt optimization for downstream generative nlp tasks. based on the availability of benchmarks and evaluation metrics, we focus on style transfer and machine translation tasks. ## results we report individual objective values as well as their expected product and average in when evaluating based on the expected product, the product method demonstrates superior performance compared to the other approaches. additionally, we observe a more balanced improvement across all rewards with volume-based methods such as hvi and product, in contrast to average and mgda. for example, in style transfer task, the "average" method improves style disproportionately higher than other objectives despite achieving the best performance based on the average metric. among the compared methods, we believe that expected product of reward best captures the balanced performance across the rewards, as it mitigates the problems of objective collapse and dominant outlier effect, described in prior sections. ## qualitative analysis in this section, we perform a qualitative analysis of the learned prompts. in moreover, we can observe some similarities and common words in high-performing prompts, demonstrating the effectiveness of certain tokens for a specific task. however, we can see that despite these similarities, the differences in the prompts from various methods can substantially affect the final evaluation results, which were shown in it is important to note that the highestperforming prompts do not necessarily need to be interpretable by humans. there is value in a discrete prompt beyond it being interpretable by humans: a discrete prompt, unlike a continuous prompt, can be passed into a black-box model like chatgpt, while a continuous prompt cannot. furthermore, discrete prompts are more likely to generalize across different llms than continuous prompts due to the common text space instead of the model-specific latent space @xcite . in applications where having an interpretable prompt is useful, corresponding rewards could be included in the multi-objective formulation. ## conclusion we empirically investigate the use of optimization techniques alongside reinforcement learning to address discrete prompt optimization in a multiobjective context. our experiments show that multi-objective methods, which directly optimize the volume, outperform those seeking monotonic update directions, achieving a better balance across all rewards.