Dynamic Multi-Reward Weighting for Multi-Style Controllable Generation (2024)

Abstract: Textual style expresses a diverse set of information, including interpersonal dynamics (e.g., formality) and the author’s emotions or attitudes (e.g., disgust). An open question is how language models can be explicitly controlled so that they weave together target styles when generating text: for example, to produce text that is both negative and non-toxic. One approach to such controlled generation is multi-objective reinforcement learning (RL), but how to best combine multiple objectives in a reward function is an open question. In this paper, we investigate various formulations of multi-style reward formulations, including calibrated outputs from discriminators and dynamic weighting by discriminator gradient magnitudes. We find that our proposed dynamic weighting outperforms static weighting approaches with respect style control while maintaining linguistic quality, and we explore its effectiveness in 2- and 3-style control.

## introduction textual style is an important component of communication that conveys information not included in the text's raw semantic content @xcite . consequently, it is vital that language models can understand and apply styles themselves. prior work has explored the domain of controlled style generation, a task in which a generative language model aims to generate text with a specified style 2 . however in practice, text frequently contains not only a single style, but a combination of styles @xcite . for example, consider being asked to give feedback to a colleague at work: both formal and positive styles would be appropriate. on the other hand, if speaking with a friend about a 1 https://github.com/minnesotanlp/ dynamic-multi-reward-weighting 2 style transfer, on the other hand, involves paraphrasing given text to a target style without altering its original meaning ## proposed method: multi-reward control in rl in this section we provide a preliminary formulation of using reinforcement learning (rl) for a single style control and then formulate the specific multi-reward control functions we propose. ## results in this section we describe our experiments for assessing the different multi-style reward formulations and the overall performance of our models for various multi-style control settings. the ques-tions we investigate are: 1. how can we most effectively combine signals from separate style discriminators into a reward function? ( § 5.1), 2. how often do two-and three-style models express the target styles in their generation, and how fluent are these generations? ( § 5.2, 5.3), and 3. which style combinations are most difficult to learn? ( § 5.4). ## discussion we propose an approach to controlled multi-style generation using reinforcement learning with a reward derived from a dynamically weighted linear combination of discriminator outputs. this novel technique results in generations that largely conform to the target styles while maintaining linguistic quality. there are multiple possible approaches to the multi-style generation problem; our approach represents only one of these. in addition to pplm, which could possibly be further optimized beyond our additions for multi-style control, there are also the other postprocessing and retraining methods discussed in section 2, along with other fine-tuning approaches (including direct preference optimization (dpo) @xcite ). in addition, this problem could be approached via prompt engineering for llms such as gpt-4. which approach is best-suited to this problem with respect to accuracy, cost, and efficiency remains an open question. future work should investigate the strongest overall approach multi-style control and the relative accuracy-efficiency trade-offs. finally, the precise theoretical relationships between individual styles is an interesting and open question. if two styles rarely co-occur, is it because their combination is impossible, or is it simply rare? does this distinction affect a language model's ability to combine the two styles? our results for the three-style models hint at the complexity of this issue. future work should consider formalizing the feasibility of different low-frequency style combinations, and encouraging language models to explore their state space more in order to uncover more rare combinations of styles. ## limitations our focus in this work is mostly centered on combining specific style dimensions, sentiment and formality, and our three-style combinations still include these specific dimensions. we make the decision to focus on specific styles due to computational limits making wider exploration infeasible, and we focus on these particular dimensions due to the high f1 scores of the discriminators for these style dimensions. however, this approach can be extended given more computing resources to further style combinations (e.g. emotional valence and arousal, honesty). it can also be extended reinforcement learning with multiple types of human feedback, as recent work investigates composing multiple types of human feedback -e.g. helpfulness, correctness -into the reward function @xcite . our approach is less effective for rare style combinations like negative and formal. for control of rare style combinations, further techniques such as initial language model fine-tuning or exploration incentives are likely needed in order to achieve the desired effect. our work also relies on discriminators which may be difficult to implement for data scarce attributes. alignment tax we also note two side effects we observed in the generations after fine-tuning. this is somewhat expected, as many researchers have documented performance on some benchmarking tasks decreasing as side effects of rl fine-tuning procedures (see e.g. @xcite ; @xcite ). first, there is a possibility that nontarget styles may also shift as a result of the finetuning process. specifically, we observe some variations in uncontrolled styles after fine-tuning (e.g. there is more joy in positive-formal than in positiveinformal; there is more anger in negative-formal than in negative-informal. these results are summarized in appendix figure 9 . second, we observe that controlling for style can alter the model's factual claims. the 500-item wikipedia subset of the evaluation set demonstrates this clearly, as the prompts (e.g. "skydio is an american...") lead the model to write a completion that makes factual assertions. see example wikipedia generations and our annotations in appendix a.4 ## ethics statement this training approach includes no explicit steering of factuality or truthfulness, which should always be accounted for before deploying llms for nonacademic use. further, it is important to be aware of the possibility that controlling for styles such as negativity or disgust can correspondingly lead to increases in toxicity and offensive speech for model generations.